Low-power fusion for negative shutter lag capture

ABSTRACT

Systems and techniques are provided for processing one or more frames. For example, a process can include obtaining a first plurality of frames associated with a first settings domain from an image capture system, wherein the first plurality of frames is captured prior to obtaining a capture input. The process can include obtaining a reference frame associated with a second settings domain from the image capture system, wherein the reference frame is captured proximate to obtaining the capture input. The process can include obtaining a second plurality of frames associated with the second settings domain from the image capture system, wherein the second plurality of frames is captured after the reference frame. The process can include, based on the reference frame, transforming at least a portion of the first plurality of frames to generate a transformed plurality of frames associated with the second settings domain.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Non-Provisional applicationSer. No. 17/476,283, filed on Sep. 15, 2021, which is herebyincorporated by reference, in its entirety and for all purposes.

FIELD

The present disclosure generally relates to the capture of images and/orvideo, and more specifically to systems and techniques for performingnegative shutter lag capture

BACKGROUND

Many devices and systems allow a scene to be captured by generatingimages (or frames) and/or video data (including multiple frames) of thescene. For example, a camera or a device including a camera can capturea sequence of frames of a scene (e.g., a video of a scene). In somecases, the sequence of frames can be processed for performing one ormore functions, can be output for display, can be output for processingand/or consumption by other devices, among other uses.

Devices (e.g., mobile devices) and systems are increasingly leveragingspecialized ultra-low power camera hardware for “always-on” (AON) camerause cases where a camera can remain on to continuously record whilemaintaining a lower power usage footprint. The AON camera can captureimages or video of unexpected events where a user may be unable toinitiate video recording prior to the event occurring. However, theoverall power consumption of AON camera setups capturing images or videocan nevertheless significantly reduce the battery life of mobiledevices, which generally have a limited battery life AON. In some cases,AON camera setups can leverage low power camera hardware for reducedpower consumption.

BRIEF SUMMARY

In some examples, systems and techniques are described for providing anegative shutter lag video and/or image capture. According to at leastone illustrative example, a method is provided for processing one ormore frames. The method includes: obtaining a first plurality of framesassociated with a first settings domain from an image capture system,wherein the first plurality of frames is captured prior to obtaining acapture input; obtaining at least one reference frame associated with asecond settings domain from the image capture system, wherein the atleast one reference frame is captured proximate to obtaining the captureinput; obtaining a second plurality of frames associated with the secondsettings domain from the image capture system, wherein the secondplurality of frames is captured after the at least one reference frame;based on the at least one reference frame, transforming at least aportion of the first plurality of frames to generate a transformedplurality of frames associated with the second settings domain.

In another example, an apparatus for processing one or more frames isprovided that includes at least one memory (e.g., configured to storedata, such as virtual content data, one or more images, etc.) and one ormore processors (e.g., implemented in circuitry) coupled to the at leastone memory. The one or more processors are configured to and can: obtaina first plurality of frames associated with a first settings domain froman image capture system, wherein the first plurality of frames iscaptured prior to obtaining a capture input; obtain at least onereference frame associated with a second settings domain from the imagecapture system, wherein the at least one reference frame is capturedproximate to obtaining the capture input; obtain a second plurality offrames associated with the second settings domain from the image capturesystem, wherein the second plurality of frames is captured after the atleast one reference frame; based on the at least one reference frame,transform at least a portion of the first plurality of frames togenerate a transformed plurality of frames associated with the secondsettings domain.

In another example, a non-transitory computer-readable medium isprovided that has stored thereon instructions that, when executed by oneor more processors, cause the one or more processors to: obtain a firstplurality of frames associated with a first settings domain from animage capture system, wherein the first plurality of frames is capturedprior to obtaining a capture input; obtain at least one reference frameassociated with a second settings domain from the image capture system,wherein the at least one reference frame is captured proximate toobtaining the capture input; obtain a second plurality of framesassociated with the second settings domain from the image capturesystem, wherein the second plurality of frames is captured after the atleast one reference frame; based on the at least one reference frame,transform at least a portion of the first plurality of frames togenerate a transformed plurality of frames associated with the secondsettings domain.

In another example, an apparatus for processing one or more frames isprovided. The apparatus includes: means for obtaining a first pluralityof frames associated with a first settings domain from an image capturesystem, wherein the first plurality of frames is captured prior toobtaining a capture input; means for obtaining at least one referenceframe associated with a second settings domain from the image capturesystem, wherein the at least one reference frame is captured proximateto obtaining the capture input; means for obtaining a second pluralityof frames associated with the second settings domain from the imagecapture system, wherein the second plurality of frames is captured afterthe at least one reference frame; means for, based on the at least onereference frame, transforming at least a portion of the first pluralityof frames to generate a transformed plurality of frames associated withthe second settings domain.

In some aspects, the first settings domain comprises a first resolutionand the second settings domain comprises a second resolution. In somecases, to transform at least the portion of the first plurality offrames, the method, apparatuses, and computer-readable medium describedabove can comprise upscaling at least the portion of the first pluralityof frames from the first resolution to the second resolution to generatethe transformed plurality of frames, wherein the transformed pluralityof frames have the second resolution.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above can further comprising: obtaining an additionalreference frame having the second resolution from the image capturesystem, wherein the additional reference frame is captured prior toobtaining the capture input, generating the upscaled plurality of frameshaving the second resolution is based on at least the portion of thefirst plurality of frames, the at least one reference frame, and theadditional reference frame, and wherein the at least one reference frameprovides a reference for upscaling at least a first portion of at leastthe portion of the first plurality of frames and the additionalreference frame provides a reference for upscaling at least a secondportion of at least the portion of the first plurality of frames.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above can further comprise: combining the transformedplurality of frames and the second plurality of frames to generate avideo associated with the second settings domain.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above can further comprise: obtaining motion informationassociated with the first plurality of frames, wherein generating thetransformed plurality of frames associated with the second settingsdomain is based on at least the portion of the first plurality offrames, the at least one reference frame, and the motion information.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above can further comprise: determining a panning directionbased on the motion information associated with the first plurality offrames; applying the panning direction to the transformed plurality offrames.

In some aspects, the first settings domain comprises a first framerateand the second settings domain comprises a second framerate. In somecases, In some cases, to transform at least the portion of the firstplurality of frames, the method, apparatuses, and computer-readablemedium described above can comprise framerate converting at least theportion of the first plurality of frames from the first framerate to thesecond framerate.

In some aspects, a first subset of the first plurality of frames iscaptured at the first framerate and a second subset of the firstplurality of frames is captured at a third framerate, different from thefirst framerate. In some cases, the third framerate is equal to or notequal to the second framerate. In some aspects, a change between thefirst framerate and the third framerate is based at least in part onmotion information associated at least one of the first subset of thefirst plurality of frames and the second subset of the first pluralityof frames.

In some aspects, the first settings domain comprises a first resolutionand a first framerate and the second settings domain comprises a secondresolution and a second framerate. In some cases, In some cases, totransform at least the portion of the first plurality of frames, themethod, apparatuses, and computer-readable medium described above cancomprise upscaling at least the portion of the first plurality of framesfrom the first resolution to the second resolution and framerateconverting at least the portion of the first plurality of frames fromthe first framerate to the second framerate.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above further comprise: obtaining an additional referenceframe associated with the second settings domain from the image capturesystem, wherein the additional reference frame is captured prior toobtaining the capture input, wherein generating the transformedplurality of frames associated with the second settings domain is basedon at least the portion of the first plurality of frames, at least onereference frame, and the additional reference frame, and wherein the atleast one reference frame provides a reference for transforming at leasta first subset of at least the portion of the first plurality of framesand the additional reference frame provides a reference for transformingat least a second subset of at least the portion of the first pluralityof frames.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above further comprise: obtaining a second reference frameassociated with the second settings domain from the image capturesystem, wherein the second reference frame is captured proximate toobtaining the capture input; based on the first reference frame,transforming at least the portion of the first plurality of frames togenerate the transformed plurality of frames associated with the secondsettings domain; and based on the second reference frame, transformingat least another portion of the first plurality of frames to generate asecond transformed plurality of frames associated with the secondsettings domain.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above further comprise: obtaining a motion estimate associatedwith the first plurality of frames; obtaining a third reference frameassociated with the second settings domain from the image capturesystem, wherein the third reference frame is captured prior to obtainingthe capture input; and based on the third reference frame, transforminga third portion of the first plurality of frames to generate a thirdtransformed plurality of frames associated with the second settingsdomain; wherein an amount of time between the first reference frame andthe third reference frame is based on the motion estimate associatedwith the first plurality of frames.

In some aspects, the first settings domain comprises at least one of afirst resolution, a first framerate, a first color depth, a first noisereduction technique, a first edge enhancement technique, a first imagestabilization technique, and a first color correction technique, and thesecond settings domain comprises at least one of a second resolution, asecond framerate, a second color depth, a second noise reductiontechnique, a second edge enhancement technique, a second imagestabilization technique, and a second color correction technique.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above further comprise: generating the transformed pluralityof frames using a trainable neural network, wherein the neural networkis trained using a training dataset comprising pairs of images, eachpair of images including a first image associated with the firstsettings domain and a second image associated with the second settingsdomain.

In some aspects, capturing the at least one reference frame proximate toobtaining the capture input comprises capturing a first availableassociated with the second settings domain after the capture input isreceived, capturing a second available frame associated with the secondsettings domain after the capture input is received, capturing a thirdavailable frame associated with the second settings domain after thecapture input is received, or capturing a fourth available frameassociated with the second settings domain after the capture input isreceived.

In some aspects, capturing the at least one reference frame proximate toobtaining the capture input comprises capturing a frame associated withthe second settings domain within 10 millisecond (ms), within 100 ms,within 500 ms, or within 1000 ms after the capture input is received.

According to at least one other example, a method is provided forprocessing one or more frames. The method includes: obtaining a firstplurality of frames associated with a first settings domain from animage capture system, wherein the first plurality of frames is capturedprior to obtaining a capture input; obtaining a reference frameassociated with a second settings domain from the image capture system,wherein the reference frame is captured proximate to obtaining thecapture input; obtaining a selection of one or more selected framesassociated with the first plurality of frames; based on the referenceframe, transforming the one or more selected frames to generate one ormore transformed frames associated with the second settings domain.

In another example, an apparatus for processing one or more frames isprovided that includes at least one memory (e.g., configured to storedata, such as virtual content data, one or more images, etc.) and one ormore processors (e.g., implemented in circuitry) coupled to the at leastone memory. The one or more processors are configured to and can: obtaina first plurality of frames associated with a first settings domain froman image capture system, wherein the first plurality of frames iscaptured prior to obtaining a capture input; obtain a reference frameassociated with a second settings domain from the image capture system,wherein the reference frame is captured proximate to obtaining thecapture input; obtain a selection of one or more selected framesassociated with the first plurality of frames; based on the referenceframe, transform the one or more selected frames to generate one or moretransformed frames associated with the second settings domain.

In another example, a non-transitory computer-readable medium isprovided that has stored thereon instructions that, when executed by oneor more processors, cause the one or more processors to: obtain a firstplurality of frames associated with a first settings domain from animage capture system, wherein the first plurality of frames is capturedprior to obtaining a capture input; obtain a reference frame associatedwith a second settings domain from the image capture system, wherein thereference frame is captured proximate to obtaining the capture input;obtain a selection of one or more selected frames associated with thefirst plurality of frames; based on the reference frame, transform theone or more selected frames to generate one or more transformed framesassociated with the second settings domain.

In another example, an apparatus for processing one or more frames isprovided. The apparatus includes: means for obtaining a first pluralityof frames associated with a first settings domain from an image capturesystem, wherein the first plurality of frames is captured prior toobtaining a capture input; means for obtaining a reference frameassociated with a second settings domain from the image capture system,wherein the reference frame is captured proximate to obtaining thecapture input; means for obtaining a selection of one or more selectedframes associated with the first plurality of frames; means for, basedon the reference frame, transforming the one or more selected frames togenerate one or more transformed frames associated with the secondsettings domain.

In some aspects, selection of one or more selected frames is based on aselection from a user interface.

In some aspects, the user interface comprises a thumbnail gallery, aslider, or a frame-by-frame review.

In some aspects, the method, apparatuses, and computer-readable mediumdescribed above further comprise: determining one or more suggestedframes from the first plurality of frames based on one or more ofdetermining whether an amount of motion or amount of change in motionexceeds a threshold, determining which frames contain interestingcontent based on a presence of one or more human faces, and determiningwhich frames contain content similar to a set of labeled set of images.

In some aspects, one or more of the apparatuses described above is or ispart of a vehicle (e.g., a computing device of a vehicle), a mobiledevice (e.g., a mobile telephone or so-called “smart phone” or othermobile device), a wearable device, an extended reality device (e.g., avirtual reality (VR) device, an augmented reality (AR) device, or amixed reality (MR) device), a personal computer, a laptop computer, aserver computer, or other device. In some aspects, an apparatus includesa camera or multiple cameras for capturing one or more images. In someaspects, the apparatus further includes a display for displaying one ormore images, notifications, and/or other displayable data. In someaspects, the apparatus can include one or more sensors, which can beused for determining a location and/or pose of the apparatuses, a stateof the apparatuses, and/or for other purposes.

This summary is not intended to identify key or essential features ofthe claimed subject matter, nor is it intended to be used in isolationto determine the scope of the claimed subject matter. The subject mattershould be understood by reference to appropriate portions of the entirespecification of this patent, any or all drawings, and each claim.

The foregoing, together with other features and embodiments, will becomemore apparent upon referring to the following specification, claims, andaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative embodiments of the present application are described indetail below with reference to the following figures:

FIG. 1 is a block diagram illustrating an architecture of an imagecapture and processing system, in accordance with some examples;

FIG. 2 is a diagram illustrating an architecture of an example extendedreality (XR) system, in accordance with some examples;

FIG. 3 is a block diagram illustrating an example image processingsystem, in accordance with some examples;

FIG. 4A is a diagram illustrating example always-on (AON) camera usescases, in accordance with some examples;

FIG. 4B is a diagram illustrating an example negative shutter lag usecase, in accordance with some examples;

FIG. 5A is a block diagram illustrating an example negative shutter lagsystem, in accordance with some examples;

FIG. 5B is a block diagram illustrating another example negative shutterlag system, in accordance with some examples;

FIG. 6 is a flow diagram illustrating an example of a process forprocessing one or more frames in accordance with some examples;

FIG. 7 is a flow diagram illustrating an example of a process forprocessing one or more frames, in accordance with some examples;

FIG. 8A is a flow diagram illustrating an example of a process forperforming a negative shutter lag capture, in accordance with someexamples;

FIG. 8B is a diagram illustrating an example of relative powerconsumption levels during the negative shutter lag capture process shownin FIG. 8A, in accordance with some examples;

FIG. 9 is a block diagram illustrating another example negative shutterlag system, in accordance with some examples;

FIG. 10 is a flow diagram illustrating another example of a process forprocessing one or more frames, in accordance with some examples;

FIG. 11 is a block diagram illustrating another example negative shutterlag system, in accordance with some examples;

FIG. 12 is a flow diagram illustrating an example of a process forprocessing one or more frames, in accordance with some examples;

FIG. 13 is a block diagram illustrating another example negative shutterlag system, in accordance with some examples;

FIG. 14 is a flow diagram illustrating is another flow diagramillustrating an example of a negative shutter lag frame capturesequence, in accordance with some examples;

FIG. 15 is a diagram illustrating an example of relative powerconsumption levels during a negative shutter lag frame capture sequence,in accordance with some examples;

FIG. 16 is a flow diagram illustrating an example of a process forprocessing one or more frames, in accordance with some examples;

FIG. 17 is a block diagram illustrating an example of a deep learningnetwork, in accordance with some examples;

FIG. 18 is a block diagram illustrating an example of a convolutionalneural network, in accordance with some examples;

FIG. 19 is a diagram illustrating an example of a computing system forimplementing certain aspects described herein.

DETAILED DESCRIPTION

Certain aspects and embodiments of this disclosure are provided below.Some of these aspects and embodiments may be applied independently andsome of them may be applied in combination as would be apparent to thoseof skill in the art. In the following description, for the purposes ofexplanation, specific details are set forth in order to provide athorough understanding of embodiments of the application. However, itwill be apparent that various embodiments may be practiced without thesespecific details. The figures and description are not intended to berestrictive.

The ensuing description provides exemplary embodiments only, and is notintended to limit the scope, applicability, or configuration of thedisclosure. Rather, the ensuing description of the exemplary embodimentswill provide those skilled in the art with an enabling description forimplementing an exemplary embodiment. It should be understood thatvarious changes may be made in the function and arrangement of elementswithout departing from the spirit and scope of the application as setforth in the appended claims.

Systems and techniques are described herein for providing a negativeshutter lag video and/or image capture system. In some examples, animage capture system can implement a lower-power or “always-on” (AON)camera that persistently or periodically operates to automaticallydetect certain objects in an environment. For instance, an image capturesystem that can capture video with an AON camera can be useful insituations where the AON camera is always pointed toward a target ofinterest. For example, one or more cameras included in a head-mounteddevice (e.g., virtual reality (VR) or augmented reality (AR)head-mounted display (HMD), AR glasses, etc.) can always be pointedwhere a user is looking, based on movement of the user's head. Whileexamples are described herein with reference to an AON camera, suchaspects can be applied to any camera or image sensor that operate in alow-power mode.

In some cases, the AON camera can leverage low power camera hardware forreduced power consumption. In some cases, the AON camera can operatewith low power settings and/or perform different or fewer imageprocessing steps for reduced power consumption. The amount of powerconsumed by the AON camera can depend upon the domain (also referred toherein as a settings domain) of the image and/or video frame capture. Insome cases, the domain of an AON camera capturing images and/or videoframes can include configuration parameters (e.g., resolution,framerate, color depth, or the like), and/or other parameters of the AONcamera. In some cases, the domain can further include the processingsteps (e.g., noise reduction, edge enhancement, image stabilization,color correction, or the like) performed on the captured images and/orvideo frames by a camera pipeline of the AON camera. In some cases,after a user initiates a video and/or image capture, a non-AON cameracan capture a still image or begin capturing video frames. The non-AONcamera can utilize higher power camera hardware and/or can operate withhigher power settings and/or different or more image processing steps ascompared to an AON or other low-power camera.

It should be understood that while specific examples of the disclosureare discussed in terms of an AON camera (or AON camera sensor) and amain camera (or main camera sensor), the systems and techniquesdescribed herein can be applied to many different camera or sensorconfigurations without departing from the scope of the presentdisclosure. In one illustrative example, a single camera or sensor canbe configured to operate with different operating modes (e.g., a lowpower AON mode and a non-AON mode). References to “AON operation” hereincan be understood to include capturing images and/or frames with one ormore AON cameras and/or operating one or more cameras in an AON mode. Inthe present disclosure, frames (e.g., video frames or images) capturedduring AON operation are sometimes referred to as low power frames(e.g., low power video frames). Similarly, references to “standardoperation” can be understood to include capturing frames with one ormore non-AON cameras and/or one or more cameras operating in a non-AONmode. In addition, non-AON cameras or sensors and/or cameras or sensorsoperating in a non-AON mode can sometimes be referred to as one or more“main cameras” or “main camera sensors”. In the present disclosure,frames (e.g., video frames) captured during standard camera operationare sometimes referred to as high power frames (e.g., high power videoframes). The systems and techniques will be described herein as beingperformed with respect to video frames. However, it will be understoodthat the systems and techniques can operate using any sequence of imagesor frames, such as consecutively captured still images.

A user can initiate an image and/or video capture by, for example,pressing a capture or record button (also referred to as a shutterbutton in some cases), performing a gesture, and/or any other method ofproviding a capture input. The time that the image capture systemreceives the capture input can be defined as time t=0. Video framescaptured during AON operation (e.g., at time t<0) can be stored inmemory (e.g., a buffer, circular buffer, video memory, or the like) thatmaintains captured video frames for a period of time (e.g., 30 seconds,1 minute, 5 minutes, etc.). The frames captured during the AON operationcan be associated with a first settings domain (hereinafter alsoreferred to as a first domain). In one illustrative example, the firstsettings domain can include a wide video graphics array (WVGA)resolution (e.g., 800 pixels×480 pixels) and a framerate of 30 framesper second (fps).

After the image capture system receives the capture input, the imagecapture system can begin capturing video frames in standard operation.In some cases, video frames captured during standard operation can beassociated with a second settings domain (hereinafter also referred toas a second domain), different from the first settings domain. In oneillustrative example, the second settings domain can include anultra-high definition (UHD) resolution (e.g., 3840 pixels×2160 pixels)and a framerate of 30 fps. The frames captured during standard operationcan include capturing a keyframe (also referred to as a reference frameherein) associated with the second settings domain (e.g., with the UHDresolution) at or proximate to t=0. As an illustrative example,capturing a keyframe proximate to the capture input can includecapturing the first available frame associated with the second settingsdomain after the capture input is received. In some cases, capturing akeyframe proximate to the capture input can include capturing the secondframe, third frame, fourth frame, or fifth frame, or any other suitableavailable frame associated with the second settings domain after thecapture input is received. In some cases, capturing a keyframe proximateto the capture input can include capturing a frame associated with thesecond settings domain within 10 millisecond (ms), within 100 ms, within500 ms, within 1000 ms, or within any other suitable time window afterthe capture input is received. If the capture input occurs in responseto an unexpected event (e.g., a child scoring a goal in soccer, a petperforming a trick, etc.), the capture input may occur after the eventthe user is interested in capturing (e.g., time t<0). In some cases, theimage processing system may have captured the event during AONoperation. In some cases, the video frames captured during AON operationassociated with the first domain can have a significantly differentappearance (e.g., different resolution, framerate, color depth,sharpness, etc.) than video frames associated with the second domain andcaptured during standard operation. In some cases, using the keyframe asa guide, the video frames captured during AON operation associated withthe first domain can be transformed to the second settings domain. Inone illustrative example, transforming video frames from the firstsettings domain to the second settings domain can include resolutionscaling, framerate scaling, changing color depth, and/or othertransformations described herein. In some cases, a composite (orstitched) video can be formed from the transformed video framesassociated with the second settings domain and the video framesassociated with the second settings domain captured after receiving thecapture input.

The process of transforming video frames from the first domain to thesecond domain using a keyframe can be referred to as a domain transformor guided domain transform. In some cases, a deep learning neuralnetwork (e.g., a domain transform model) can be trained to perform theguided domain transform at least in part by converting a first portionof video frames associated with the second domain (e.g., from a videoincluded in a training data set) to the first domain, providing akeyframe associated with the second domain from the original video, andtransforming the first portion of video frames back to the second domainusing the keyframe as a guide. The resulting transformed video framescan be directly compared to the original video frames and a lossfunction can be used to determine an amount of error between thetransformed video frames and the original frames. The parameters (e.g.,weights, biases, etc.) of the deep learning network can be adjusted (ortuned) based on the error. Such a training process can be referred to assupervised learning using backpropagation, which can be performed untilthe tuned parameters provide a desired result. In some cases, the domaintransform model can be trained utilizing a deep generative neuralnetwork model (e.g., generative adversarial network (GAN)).

In some cases, AON operation can include capturing video frame data at alow resolution (e.g., VGA resolution of 640 pixel×480 pixels or WVGAresolution). In some cases, standard operation can include capturingvideo data at a higher resolution than AON operation (e.g., 720p, 1080i,1080p, 4K, 8K or the like). Capturing and processing video data at thelower resolution during AON operation can consume less power thancapturing and processing video data at the higher resolution duringstandard operation. For example, the amount of power required to readthe video frame data captured by the image capture system can increaseas the number of pixels read out increases. In addition, the amount ofpower required for image post-processing of the captured video framesdata can also increase as the number of pixels in the video framesincrease. In some cases, the low resolution video frames can be upscaledto the higher resolution. The example of a low resolution capture duringAON operation and higher resolution capture during standard operationdescribed above provides one illustrative example of different domainsthat can be used for capturing video frames in a negative shutter lagsystem.

The process of upscaling using a high resolution keyframe can bereferred to as a guided super-resolution process. A guidedsuper-resolution process is an illustrative example of a guided domaintransform. In some cases, a deep learning neural network (e.g., a guidedsuper-resolution model) can be trained to perform the guidedsuper-resolution process in a training process similar to the processfor training a domain transform model described above. For example, theguided super resolution model can be trained at least in part bydownscaling a portion of video frames from a high resolution video(e.g., a video included in a training data set), providing a highresolution keyframe from the high resolution video, and upscaling thedownscaled portion of the video frames from the high resolution videoback to the original resolution using the high resolution keyframe as aguide. The upscaled video frames can be directly compared to theoriginal high resolution video frames and a loss function can be used todetermine an amount of error between the upscaled video frames and theoriginal high resolution video frames of the video. The parameters(e.g., weights, biases, etc.) of the deep learning network can beadjusted (or tuned) based on the error. Such a training process can bereferred to as supervised learning using backpropagation, which can beperformed until the tuned parameters provide a desired result. In somecases, the guided super-resolution model can be trained utilizing a GAN.

In some cases, the image capture system operating in the AON mode cancapture video having a lower framerate than the video captured duringthe high resolution mode. By reducing the framerate and/or resolutionduring the AON mode, the image capture system can operate with lowerpower consumption during the AON mode when compared to the highresolution mode. In some cases, additional power can be saved in the AONmode by utilizing an adaptive framerate. In some implementations, theimage capture system can utilize inertial motion estimation from aninertial sensor to determine an amount of motion of the image capturesystem (e.g., motion of a head mounted device). In some implementations,the image capture system can perform optical motion estimation (e.g., byanalyzing the captured video frames) to determine the amount of motionof the image capture system. If the image capture system is still or hasonly a small amount of motion, the framerate for capturing video framescan be set to a lower framerate setting (e.g., to 15 fps). On the otherhand, if the image capture system and/or the scene being captures has alarge amount of motion, the framerate for capturing video frames duringAON operation can be set to a high framerate setting (e.g., 30 fps, 60fps). In some cases, the maximum framerate during the AON mode can beequal to the framerate setting for the high resolution mode. In somecases, in addition to storing the video frames in memory during the AONmode, information indicating the framerate at which the video frameswere captured can also be stored.

In some implementations, in addition to capturing a keyframe at themoment a capture input is received (e.g., at t=0) or proximate to thecapture input, the image capture system can capture additional highresolution keyframes periodically during AON operation. As anillustrative example, capturing a keyframe proximate to the captureinput can include capturing the first available high resolution frameafter the capture input is received. In some cases, capturing a keyframeproximate to the capture input can include capturing the second, third,fourth, or fifth high resolution available frame after the capture inputis received. In some cases, capturing a keyframe proximate to thecapture input can include capturing a high resolution frame within 10ms, within 100 ms, within 500 ms, within 1000 ms, or within any othersuitable time window after the capture input is received. For example, akeyframe can be captured every half second, every second, every fiveseconds, every ten seconds, or any other suitable duration. Theadditional high resolution keyframes captured during AON operation canbe stored in memory (e.g., a buffer) along with the low resolution videoframes. Each high resolution keyframe that is captured can be used as aguide for video frames captured during AON operation that are close intime to the keyframe. In some cases, the additional high resolutionkeyframes can improve the guided super-resolution process where theimage capture system is moving over time and/or objects in the scene aremoving over time. By periodically capturing high resolution keyframes,the scene captured in each keyframe is more likely to capture at least aportion of the same scene captured during AON operation within a certaintime period (e.g., within half a second, one second, five seconds, orten seconds). In some cases, the capture rate for high resolutionkeyframes can be determined based on an amount of motion of the imagecapture system. In some cases, the amount of motion can be determinedbased on readings from an inertial motion sensor and/or from performingoptical motion detection on the video frames captured during AONoperation.

In some cases, inertial sensor data can also be stored in memory alongwith video frames captured during the AON mode and utilized in theguided super-resolution process. For example, if data from the inertialsensors captured during the AON mode indicates the image capture systemwas moving to the left, the guided super-resolution can utilize theinformation about the motion in the scene to ensure that, for example,the video that it outputs pans to the left appropriately to match themeasured motion.

While specific example domain transforms (e.g., resolution upscaling,framerate adjustment, and color depth adjustment) are described hereinfor performing negative shutter lag capture, the systems and techniquesdescribed herein can be used to perform negative shutter lag capturewith other domain transforms without departing from the scope of thepresent disclosure. For example, a negative shutter lag system cancapture monochrome video frames during AON operation and color videoframes during standard operation. In such an example, the domaintransform can include colorizing the monochrome frames.

Various aspects of the techniques described herein will be discussedbelow with respect to the figures. FIG. 1 is a block diagramillustrating an architecture of an image capture and processing system100. The image capture and processing system 100 includes variouscomponents that are used to capture and process images of scenes (e.g.,an image of a scene 110). The image capture and processing system 100can capture standalone images (or photographs) and/or can capture videosthat include multiple images (or video frames) in a particular sequence.A lens 115 of the system 100 faces a scene 110 and receives light fromthe scene 110. The lens 115 bends the light toward the image sensor 130.The light received by the lens 115 passes through an aperture controlledby one or more control mechanisms 120 and is received by the imagesensor 130.

The one or more control mechanisms 120 may control exposure, focus,and/or zoom based on information from the image sensor 130 and/or basedon information from the image processor 150. The one or more controlmechanisms 120 may include multiple mechanisms and components; forinstance, the control mechanisms 120 may include one or more exposurecontrol mechanisms 125A, one or more focus control mechanisms 125B,and/or one or more zoom control mechanisms 125C. The one or more controlmechanisms 120 may also include additional control mechanisms besidesthose that are illustrated, such as control mechanisms controllinganalog gain, flash, HDR, depth of field, and/or other image captureproperties.

The focus control mechanism 125B of the control mechanisms 120 canobtain a focus setting. In some examples, focus control mechanism 125Bstore the focus setting in a memory register. Based on the focussetting, the focus control mechanism 125B can adjust the position of thelens 115 relative to the position of the image sensor 130. For example,based on the focus setting, the focus control mechanism 125B can movethe lens 115 closer to the image sensor 130 or farther from the imagesensor 130 by actuating a motor or servo (or other lens mechanism),thereby adjusting focus. In some cases, additional lenses may beincluded in the system 100, such as one or more microlenses over eachphotodiode of the image sensor 130, which each bend the light receivedfrom the lens 115 toward the corresponding photodiode before the lightreaches the photodiode. The focus setting may be determined via contrastdetection autofocus (CDAF), phase detection autofocus (PDAF), hybridautofocus (HAF), or some combination thereof. The focus setting may bedetermined using the control mechanism 120, the image sensor 130, and/orthe image processor 150. The focus setting may be referred to as animage capture setting and/or an image processing setting.

The exposure control mechanism 125A of the control mechanisms 120 canobtain an exposure setting. In some cases, the exposure controlmechanism 125A stores the exposure setting in a memory register. Basedon this exposure setting, the exposure control mechanism 125A cancontrol a size of the aperture (e.g., aperture size or f/stop), aduration of time for which the aperture is open (e.g., exposure time orshutter speed), a sensitivity of the image sensor 130 (e.g., ISO speedor film speed), analog gain applied by the image sensor 130, or anycombination thereof. The exposure setting may be referred to as an imagecapture setting and/or an image processing setting.

The zoom control mechanism 125C of the control mechanisms 120 can obtaina zoom setting. In some examples, the zoom control mechanism 125C storesthe zoom setting in a memory register. Based on the zoom setting, thezoom control mechanism 125C can control a focal length of an assembly oflens elements (lens assembly) that includes the lens 115 and one or moreadditional lenses. For example, the zoom control mechanism 125C cancontrol the focal length of the lens assembly by actuating one or moremotors or servos (or other lens mechanism) to move one or more of thelenses relative to one another. The zoom setting may be referred to asan image capture setting and/or an image processing setting. In someexamples, the lens assembly may include a parfocal zoom lens or avarifocal zoom lens. In some examples, the lens assembly may include afocusing lens (which can be lens 115 in some cases) that receives thelight from the scene 110 first, with the light then passing through anafocal zoom system between the focusing lens (e.g., lens 115) and theimage sensor 130 before the light reaches the image sensor 130. Theafocal zoom system may, in some cases, include two positive (e.g.,converging, convex) lenses of equal or similar focal length (e.g.,within a threshold difference of one another) with a negative (e.g.,diverging, concave) lens between them. In some cases, the zoom controlmechanism 125C moves one or more of the lenses in the afocal zoomsystem, such as the negative lens and one or both of the positivelenses.

The image sensor 130 includes one or more arrays of photodiodes or otherphotosensitive elements. Each photodiode measures an amount of lightthat eventually corresponds to a particular pixel in the image producedby the image sensor 130. In some cases, different photodiodes may becovered by different color filters, and may thus measure light matchingthe color of the filter covering the photodiode. For instance, Bayercolor filters include red color filters, blue color filters, and greencolor filters, with each pixel of the image generated based on red lightdata from at least one photodiode covered in a red color filter, bluelight data from at least one photodiode covered in a blue color filter,and green light data from at least one photodiode covered in a greencolor filter. Other types of color filters may use yellow, magenta,and/or cyan (also referred to as “emerald”) color filters instead of orin addition to red, blue, and/or green color filters. Some image sensors(e.g., image sensor 130) may lack color filters altogether, and mayinstead use different photodiodes throughout the pixel array (in somecases vertically stacked). The different photodiodes throughout thepixel array can have different spectral sensitivity curves, thereforeresponding to different wavelengths of light. Monochrome image sensorsmay also lack color filters and therefore lack color depth.

In some cases, the image sensor 130 may alternately or additionallyinclude opaque and/or reflective masks that block light from reachingcertain photodiodes, or portions of certain photodiodes, at certaintimes and/or from certain angles, which may be used for phase detectionautofocus (PDAF). The image sensor 130 may also include an analog gainamplifier to amplify the analog signals output by the photodiodes and/oran analog to digital converter (ADC) to convert the analog signalsoutput of the photodiodes (and/or amplified by the analog gainamplifier) into digital signals. In some cases, certain components orfunctions discussed with respect to one or more of the controlmechanisms 120 may be included instead or additionally in the imagesensor 130. The image sensor 130 may be a charge-coupled device (CCD)sensor, an electron-multiplying CCD (EMCCD) sensor, an active-pixelsensor (APS), a complimentary metal-oxide semiconductor (CMOS), anN-type metal-oxide semiconductor (NMOS), a hybrid CCD/CMOS sensor (e.g.,sCMOS), or some other combination thereof.

The image processor 150 may include one or more processors, such as oneor more image signal processors (ISPs) (including ISP 154), one or morehost processors (including host processor 152), and/or one or more ofany other type of processor 1910 discussed with respect to the computingsystem 1900. The host processor 152 can be a digital signal processor(DSP) and/or other type of processor. In some implementations, the imageprocessor 150 is a single integrated circuit or chip (e.g., referred toas a system-on-chip or SoC) that includes the host processor 152 and theISP 154. In some cases, the chip can also include one or moreinput/output ports (e.g., input/output (I/O) ports 156), centralprocessing units (CPUs), graphics processing units (GPUs), broadbandmodems (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components(e.g., Bluetooth™, Global Positioning System (GPS), etc.), anycombination thereof, and/or other components. The I/O ports 156 caninclude any suitable input/output ports or interface according to one ormore protocol or specification, such as an Inter-Integrated Circuit 2(I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a SerialPeripheral Interface (SPI) interface, a serial General PurposeInput/Output (GPIO) interface, a Mobile Industry Processor Interface(MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, anAdvanced High-performance Bus (AHB) bus, any combination thereof, and/orother input/output port. In one illustrative example, the host processor152 can communicate with the image sensor 130 using an I2C port, and theISP 154 can communicate with the image sensor 130 using a MIPI port.

The image processor 150 may perform a number of tasks, such asde-mosaicing, color space conversion, image frame downsampling, pixelinterpolation, automatic exposure (AE) control, automatic gain control(AGC), CDAF, PDAF, automatic white balance, merging of image frames toform an HDR image, image recognition, object recognition, featurerecognition, receipt of inputs, managing outputs, managing memory, orsome combination thereof. The image processor 150 may store image framesand/or processed images in random access memory (RAM) 140/1925,read-only memory (ROM) 145/1920, a cache, a memory unit, another storagedevice, or some combination thereof.

Various input/output (I/O) devices 160 may be connected to the imageprocessor 150. The I/O devices 160 can include a display screen, akeyboard, a keypad, a touchscreen, a trackpad, a touch-sensitivesurface, a printer, any other output devices 1935, any other inputdevices 1945, or some combination thereof. In some cases, a caption maybe input into the image processing device 105B through a physicalkeyboard or keypad of the I/O devices 160, or through a virtual keyboardor keypad of a touchscreen of the I/O devices 160. The I/O 160 mayinclude one or more ports, jacks, or other connectors that enable awired connection between the system 100 and one or more peripheraldevices, over which the system 100 may receive data from the one or moreperipheral devices and/or transmit data to the one or more peripheraldevices. The I/O 160 may include one or more wireless transceivers thatenable a wireless connection between the system 100 and one or moreperipheral devices, over which the system 100 may receive data from theone or more peripheral devices and/or transmit data to the one or moreperipheral devices. The peripheral devices may include any of thepreviously-discussed types of I/O devices 160 and may themselves beconsidered I/O devices 160 once they are coupled to the ports, jacks,wireless transceivers, or other wired and/or wireless connectors.

In some cases, the image capture and processing system 100 may be asingle device. In some cases, the image capture and processing system100 may be two or more separate devices, including an image capturedevice 105A (e.g., a camera) and an image processing device 105B (e.g.,a computing device coupled to the camera). In some implementations, theimage capture device 105A and the image processing device 105B may becoupled together, for example via one or more wires, cables, or otherelectrical connectors, and/or wirelessly via one or more wirelesstransceivers. In some implementations, the image capture device 105A andthe image processing device 105B may be disconnected from one another.

As shown in FIG. 1 , a vertical dashed line divides the image captureand processing system 100 of FIG. 1 into two portions that represent theimage capture device 105A and the image processing device 105B,respectively. The image capture device 105A includes the lens 115,control mechanisms 120, and the image sensor 130. The image processingdevice 105B includes the image processor 150 (including the ISP 154 andthe host processor 152), the RAM 140, the ROM 145, and the I/O 160. Insome cases, certain components illustrated in the image capture device105A, such as the ISP 154 and/or the host processor 152, may be includedin the image capture device 105A.

The image capture and processing system 100 can include an electronicdevice, such as a mobile or stationary telephone handset (e.g.,smartphone, cellular telephone, or the like), a desktop computer, alaptop or notebook computer, a tablet computer, a set-top box, atelevision, a camera, a display device, a digital media player, a videogaming console, a video streaming device, an Internet Protocol (IP)camera, or any other suitable electronic device. In some examples, theimage capture and processing system 100 can include one or more wirelesstransceivers for wireless communications, such as cellular networkcommunications, 802.11 wi-fi communications, wireless local area network(WLAN) communications, or some combination thereof. In someimplementations, the image capture device 105A and the image processingdevice 105B can be different devices. For instance, the image capturedevice 105A can include a camera device and the image processing device105B can include a computing device, such as a mobile handset, a desktopcomputer, or other computing device.

While the image capture and processing system 100 is shown to includecertain components, one of ordinary skill will appreciate that the imagecapture and processing system 100 can include more components than thoseshown in FIG. 1 . The components of the image capture and processingsystem 100 can include software, hardware, or one or more combinationsof software and hardware. For example, in some implementations, thecomponents of the image capture and processing system 100 can includeand/or can be implemented using electronic circuits or other electronichardware, which can include one or more programmable electronic circuits(e.g., microprocessors, GPUs, DSPs, CPUs, and/or other suitableelectronic circuits), and/or can include and/or be implemented usingcomputer software, firmware, or any combination thereof, to perform thevarious operations described herein. The software and/or firmware caninclude one or more instructions stored on a computer-readable storagemedium and executable by one or more processors of the electronic deviceimplementing the image capture and processing system 100.

In some examples, the extended reality (XR) system 200 of FIG. 2 caninclude the image capture and processing system 100, the image capturedevice 105A, the image processing device 105B, or a combination thereof.

FIG. 2 is a diagram illustrating an architecture of an XR system 200, inaccordance with some aspects of the disclosure. The XR system 200 canrun (or execute) XR applications and implement XR operations. In someexamples, the XR system 200 can perform tracking and localization,mapping of an environment in the physical world (e.g., a scene), and/orpositioning and rendering of virtual content on a display 209 (e.g., ascreen, visible plane/region, and/or other display) as part of an XRexperience. For example, the XR system 200 can generate a map (e.g., athree-dimensional (3D) map) of an environment in the physical world,track a pose (e.g., location and position) of the XR system 200 relativeto the environment (e.g., relative to the 3D map of the environment),position and/or anchor virtual content in a specific location(s) on themap of the environment, and render the virtual content on the display209 such that the virtual content appears to be at a location in theenvironment corresponding to the specific location on the map of thescene where the virtual content is positioned and/or anchored. Thedisplay 209 can include a glass, a screen, a lens, a projector, and/orother display mechanism that allows a user to see the real-worldenvironment and also allows XR content to be overlaid, overlapped,blended with, or otherwise displayed thereon.

In this illustrative example, the XR system 200 includes one or moreimage sensors 202, an accelerometer 204, a gyroscope 206, storage 207,compute components 210, an XR engine 220, an interface layout and inputmanagement engine 222, an image processing engine 224, and a renderingengine 226. It should be noted that the components 202-226 shown in FIG.2 are non-limiting examples provided for illustrative and explanationpurposes, and other examples can include more, less, or differentcomponents than those shown in FIG. 2 . For example, in some cases, theXR system 200 can include one or more other sensors (e.g., one or moreinertial measurement units (IMUs), radars, light detection and ranging(LIDAR) sensors, radio detection and ranging (RADAR) sensors, sounddetection and ranging (SODAR) sensors, sound navigation and ranging(SONAR) sensors, audio sensors, etc.), one or more display devices, onemore other processing engines, one or more other hardware components,and/or one or more other software and/or hardware components that arenot shown in FIG. 2 . While various components of the XR system 200,such as the image sensor 202, may be referenced in the singular formherein, it should be understood that the XR system 200 may includemultiple of any component discussed herein (e.g., multiple image sensors202).

The XR system 200 includes or is in communication with (wired orwirelessly) an input device 208. The input device 208 can include anysuitable input device, such as a touchscreen, a pen or other pointerdevice, a keyboard, a mouse, a button or key, a microphone for receivingvoice commands, a gesture input device for receiving gesture commands, avideo game controller, a steering wheel, a joystick, a set of buttons, atrackball, a remote control, any other input device 1945 discussedherein, or any combination thereof. In some cases, the image sensor 202can capture images that can be processed for interpreting gesturecommands.

In some implementations, the one or more image sensors 202, theaccelerometer 204, the gyroscope 206, storage 207, compute components210, XR engine 220, interface layout and input management engine 222,image processing engine 224, and rendering engine 226 can be part of thesame computing device. For example, in some cases, the one or more imagesensors 202, the accelerometer 204, the gyroscope 206, storage 207,compute components 210, XR engine 220, interface layout and inputmanagement engine 222, image processing engine 224, and rendering engine226 can be integrated into a head-mounted display (HMD), extendedreality glasses, smartphone, laptop, tablet computer, gaming system,and/or any other computing device. However, in some implementations, theone or more image sensors 202, the accelerometer 204, the gyroscope 206,storage 207, compute components 210, XR engine 220, interface layout andinput management engine 222, image processing engine 224, and renderingengine 226 can be part of two or more separate computing devices. Forexample, in some cases, some of the components 202-226 can be part of,or implemented by, one computing device and the remaining components canbe part of, or implemented by, one or more other computing devices.

The storage 207 can be any storage device(s) for storing data. Moreover,the storage 207 can store data from any of the components of the XRsystem 200. For example, the storage 207 can store data from the imagesensor 202 (e.g., image or video data), data from the accelerometer 204(e.g., measurements), data from the gyroscope 206 (e.g., measurements),data from the compute components 210 (e.g., processing parameters,preferences, virtual content, rendering content, scene maps, trackingand localization data, object detection data, privacy data, XRapplication data, face recognition data, occlusion data, etc.), datafrom the XR engine 220, data from the interface layout and inputmanagement engine 222, data from the image processing engine 224, and/ordata from the rendering engine 226 (e.g., output frames). In someexamples, the storage 207 can include a buffer for storing frames forprocessing by the compute components 210.

The one or more compute components 210 can include a central processingunit (CPU) 212, a graphics processing unit (GPU) 214, a digital signalprocessor (DSP) 216, an image signal processor (ISP) 218, and/or otherprocessor (e.g., a neural processing unit (NPU) implementing one or moretrained neural networks). The compute components 210 can perform variousoperations such as image enhancement, computer vision, graphicsrendering, extended reality operations (e.g., tracking, localization,pose estimation, mapping, content anchoring, content rendering, etc.),image and/or video processing, sensor processing, recognition (e.g.,text recognition, facial recognition, object recognition, featurerecognition, tracking or pattern recognition, scene recognition,occlusion detection, etc.), trained machine learning operations,filtering, and/or any of the various operations described herein. Insome examples, the compute components 210 can implement (e.g., control,operate, etc.) the XR engine 220, the interface layout and inputmanagement engine 222, the image processing engine 224, and therendering engine 226. In other examples, the compute components 210 canalso implement one or more other processing engines.

The image sensor 202 can include any image and/or video sensors orcapturing devices. In some examples, the image sensor 202 can be part ofa multiple-camera assembly, such as a dual-camera assembly. The imagesensor 202 can capture image and/or video content (e.g., raw imageand/or video data), which can then be processed by the computecomponents 210, the XR engine 220, the interface layout and inputmanagement engine 222, the image processing engine 224, and/or therendering engine 226 as described herein. In some examples, the imagesensors 202 may include an image capture and processing system 100, animage capture device 105A, an image processing device 105B, or acombination thereof.

In some examples, the image sensor 202 can capture image data and cangenerate images (also referred to as frames) based on the image dataand/or can provide the image data or frames to the XR engine 220, theinterface layout and input management engine 222, the image processingengine 224, and/or the rendering engine 226 for processing. An image orframe can include a video frame of a video sequence or a still image. Animage or frame can include a pixel array representing a scene. Forexample, an image can be a red-green-blue (RGB) image having red, green,and blue color components per pixel; a luma, chroma-red, chroma-blue(YCbCr) image having a luma component and two chroma (color) components(chroma-red and chroma-blue) per pixel; or any other suitable type ofcolor or monochrome image.

In some cases, the image sensor 202 (and/or other camera of the XRsystem 200) can be configured to also capture depth information. Forexample, in some implementations, the image sensor 202 (and/or othercamera) can include an RGB-depth (RGB-D) camera. In some cases, the XRsystem 200 can include one or more depth sensors (not shown) that areseparate from the image sensor 202 (and/or other camera) and that cancapture depth information. For instance, such a depth sensor can obtaindepth information independently from the image sensor 202. In someexamples, a depth sensor can be physically installed in the same generallocation as the image sensor 202, but may operate at a differentfrequency or framerate from the image sensor 202. In some examples, adepth sensor can take the form of a light source that can project astructured or textured light pattern, which may include one or morenarrow bands of light, onto one or more objects in a scene. Depthinformation can then be obtained by exploiting geometrical distortionsof the projected pattern caused by the surface shape of the object. Inone example, depth information may be obtained from stereo sensors suchas a combination of an infra-red structured light projector and aninfra-red camera registered to a camera (e.g., an RGB camera).

The XR system 200 can also include other sensors in its one or moresensors. The one or more sensors can include one or more accelerometers(e.g., accelerometer 204), one or more gyroscopes (e.g., gyroscope 206),and/or other sensors. The one or more sensors can provide velocity,orientation, and/or other position-related information to the computecomponents 210. For example, the accelerometer 204 can detectacceleration by the XR system 200 and can generate accelerationmeasurements based on the detected acceleration. In some cases, theaccelerometer 204 can provide one or more translational vectors (e.g.,up/down, left/right, forward/back) that can be used for determining aposition or pose of the XR system 200. The gyroscope 206 can detect andmeasure the orientation and angular velocity of the XR system 200. Forexample, the gyroscope 206 can be used to measure the pitch, roll, andyaw of the XR system 200. In some cases, the gyroscope 206 can provideone or more rotational vectors (e.g., pitch, yaw, roll). In someexamples, the image sensor 202 and/or the XR engine 220 can usemeasurements obtained by the accelerometer 204 (e.g., one or moretranslational vectors) and/or the gyroscope 206 (e.g., one or morerotational vectors) to calculate the pose of the XR system 200. Aspreviously noted, in other examples, the XR system 200 can also includeother sensors, such as an inertial measurement unit (IMU), amagnetometer, a gaze and/or eye tracking sensor, a machine visionsensor, a smart scene sensor, a speech recognition sensor, an impactsensor, a shock sensor, a position sensor, a tilt sensor, etc.

As noted above, in some cases, the one or more sensors can include atleast one IMU. An IMU is an electronic device that measures the specificforce, angular rate, and/or the orientation of the XR system 200, usinga combination of one or more accelerometers, one or more gyroscopes,and/or one or more magnetometers. In some examples, the one or moresensors can output measured information associated with the capture ofan image captured by the image sensor 202 (and/or other camera of the XRsystem 200) and/or depth information obtained using one or more depthsensors of the XR system 200.

The output of one or more sensors (e.g., the accelerometer 204, thegyroscope 206, one or more IMUs, and/or other sensors) can be used bythe XR engine 220 to determine a pose of the XR system 200 (alsoreferred to as the head pose) and/or the pose of the image sensor 202(or other camera of the XR system 200). In some cases, the pose of theXR system 200 and the pose of the image sensor 202 (or other camera) canbe the same. The pose of image sensor 202 refers to the position andorientation of the image sensor 202 relative to a frame of reference(e.g., with respect to the object detected by the image sensor 202). Insome implementations, the camera pose can be determined for 6-Degrees OfFreedom (6DoF), which refers to three translational components (e.g.,which can be given by X (horizontal), Y (vertical), and Z (depth)coordinates relative to a frame of reference, such as the image plane)and three angular components (e.g. roll, pitch, and yaw relative to thesame frame of reference). In some implementations, the camera pose canbe determined for 3-Degrees Of Freedom (3DoF), which refers to the threeangular components (e.g. roll, pitch, and yaw).

In some cases, a device tracker (not shown) can use the measurementsfrom the one or more sensors and image data from the image sensor 202 totrack a pose (e.g., a 6DoF pose) of the XR system 200. For example, thedevice tracker can fuse visual data (e.g., using a visual trackingsolution) from the image data with inertial data from the measurementsto determine a position and motion of the XR system 200 relative to thephysical world (e.g., the scene) and a map of the physical world. Asdescribed below, in some examples, when tracking the pose of the XRsystem 200, the device tracker can generate a three-dimensional (3D) mapof the scene (e.g., the real world) and/or generate updates for a 3D mapof the scene. The 3D map updates can include, for example and withoutlimitation, new or updated features and/or feature or landmark pointsassociated with the scene and/or the 3D map of the scene, localizationupdates identifying or updating a position of the XR system 200 withinthe scene and the 3D map of the scene, etc. The 3D map can provide adigital representation of a scene in the real/physical world. In someexamples, the 3D map can anchor location-based objects and/or content toreal-world coordinates and/or objects. The XR system 200 can use amapped scene (e.g., a scene in the physical world represented by, and/orassociated with, a 3D map) to merge the physical and virtual worldsand/or merge virtual content or objects with the physical environment.

In some aspects, the pose of image sensor 202 and/or the XR system 200as a whole can be determined and/or tracked by the compute components210 using a visual tracking solution based on images captured by theimage sensor 202 (and/or other camera of the XR system 200). Forinstance, in some examples, the compute components 210 can performtracking using computer vision-based tracking, model-based tracking,and/or simultaneous localization and mapping (SLAM) techniques. Forinstance, the compute components 210 can perform SLAM or can be incommunication (wired or wireless) with a SLAM system (not shown). SLAMrefers to a class of techniques where a map of an environment (e.g., amap of an environment being modeled by XR system 200) is created whilesimultaneously tracking the pose of a camera (e.g., image sensor 202)and/or the XR system 200 relative to that map. The map can be referredto as a SLAM map and can be three-dimensional (3D). The SLAM techniquescan be performed using color or grayscale image data captured by theimage sensor 202 (and/or other camera of the XR system 200), and can beused to generate estimates of 6DoF pose measurements of the image sensor202 and/or the XR system 200. Such a SLAM technique configured toperform 6DoF tracking can be referred to as 6DoF SLAM. In some cases,the output of the one or more sensors (e.g., the accelerometer 204, thegyroscope 206, one or more IMUs, and/or other sensors) can be used toestimate, correct, and/or otherwise adjust the estimated pose.

In some cases, the 6DoF SLAM (e.g., 6DoF tracking) can associatefeatures observed from certain input images from the image sensor 202(and/or other camera) to the SLAM map. For example, 6DoF SLAM can usefeature point associations from an input image to determine the pose(position and orientation) of the image sensor 202 and/or XR system 200for the input image. 6DoF mapping can also be performed to update theSLAM map. In some cases, the SLAM map maintained using the 6DoF SLAM cancontain 3D feature points triangulated from two or more images. Forexample, keyframes can be selected from input images or a video streamto represent an observed scene. For every keyframe, a respective 6DoFcamera pose associated with the keyframe can be determined. The pose ofthe image sensor 202 and/or the XR system 200 can be determined byprojecting features from the 3D SLAM map into an image or video frameand updating the camera pose from verified 2D-3D correspondences.

In one illustrative example, the compute components 210 can extractfeature points from certain input images (e.g., every input image, asubset of the input images, etc.) or from each keyframe. A feature point(also referred to as a registration point) as used herein is adistinctive or identifiable part of an image, such as a part of a hand,an edge of a table, among others. Features extracted from a capturedimage can represent distinct feature points along three-dimensionalspace (e.g., coordinates on X, Y, and Z-axes), and every feature pointcan have an associated feature location. The feature points in keyframeseither match (are the same or correspond to) or fail to match thefeature points of previously-captured input images or keyframes. Featuredetection can be used to detect the feature points. Feature detectioncan include an image processing operation used to examine one or morepixels of an image to determine whether a feature exists at a particularpixel. Feature detection can be used to process an entire captured imageor certain portions of an image. For each image or keyframe, oncefeatures have been detected, a local image patch around the feature canbe extracted. Features may be extracted using any suitable technique,such as Scale Invariant Feature Transform (SIFT) (which localizesfeatures and generates their descriptions), Learned Invariant FeatureTransform (LIFT), Speed Up Robust Features (SURF), GradientLocation-Orientation histogram (GLOH), Oriented Fast and Rotated Brief(ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast RetinaKeypoint (FREAK), KAZE, Accelerated KAZE (AKAZE), Normalized CrossCorrelation (NCC), descriptor matching, another suitable technique, or acombination thereof.

In some cases, the XR system 200 can also track the hand and/or fingersof the user to allow the user to interact with and/or control virtualcontent in a virtual environment. For example, the XR system 200 cantrack a pose and/or movement of the hand and/or fingertips of the userto identify or translate user interactions with the virtual environment.The user interactions can include, for example and without limitation,moving an item of virtual content, resizing the item of virtual content,selecting an input interface element in a virtual user interface (e.g.,a virtual representation of a mobile phone, a virtual keyboard, and/orother virtual interface), providing an input through a virtual userinterface, etc.

FIG. 3 illustrates an example block diagram of an image processingsystem 300. In some cases, the image processing system 300 can include,or can be included in, an image capture and processing system 100, animage capture device 105A, an image processing device 105B, an XR system200, portions thereof, or any combination thereof. In the illustrativeexample of FIG. 3 , the image processing system 300 includes AON cameraprocessing subsystem 302, a main camera processing subsystem 304, agraphic processing subsystem 306, a video processing subsystem 308, acentral processing unit (CPU) 310, a DRAM subsystem 312, and an SRAM320.

In some implementations, the AON camera processing subsystem 302 canreceive inputs from an AON camera sensor 316 and the main cameraprocessing subsystem 304 can receive inputs from a main camera sensor318. The AON camera sensor 316 and the main camera sensor 318 caninclude any image and/or video sensors or capturing devices. In somecases, the AON camera sensor 316 and the main camera sensor 318 can bepart of a multiple-camera assembly, such as a dual-camera assembly. Insome examples, the AON camera sensor 316 and the main camera sensor 318may include an image capture and processing system 100, an image capturedevice 105A, an image processing device 105B, or a combination thereof.In some implementations, the AON camera processing subsystem 302 of theimage processing system 300 can communicate with the AON camera sensor316 to send and/or receive operational parameters to/from the AON camerasensor 316. Similarly, in some implementations, the main cameraprocessing subsystem 304 of the image processing system 300 cancommunicate with the main camera sensor 318 to send and/or receiveoperational parameters to/from the main camera sensor 318. The DRAMsubsystem 312 of the image processing system 300 can communicate withDRAM 314 over a data bus 315. For example, the DRAM subsystem 312 cansend video frames to and/or retrieve video frames from the DRAM 314. Insome implementations, the image processing system 300 can include alocal SRAM 320.

In some cases, the AON camera sensor 316 can include optimizations forreducing power consumption. In some cases, the AON camera processingsubsystem 302 can be configured to store data (e.g., video frame data)in SRAM 320 located within the image processing system 300. In somecases, storing data in SRAM 320 can conserve power by reducing the powerrequired to drive data and address lines when compared to drivingsignals over the data bus 315 to communicate with DRAM 314. In someimplementations, island voltage rails can be used to power the AONcamera sensor 316 and AON camera processing subsystem 302. In somecases, using island rails can conserve power by preventing inactivecomponents of the image processing system 300 from drawing power. Insome examples, the AON camera sensor 316 can be clocked with a low powerclock source such as one or more ring oscillators. In someimplementations, the image and/or video captured by the AON camerasensor 316 can be associated with a different domain from the imagesand/or video captured by the main camera sensor 318. As described above,a domain can include, without limitation, characteristics or parametersof the frames captured by a camera sensor such as resolution, colordepth, and/or framerate. In one illustrative example, the AON camerasensor 316 can capture images with a lower resolution than the maincamera sensor 318. In some cases, capturing lower resolution frames withthe AON camera sensor 316 can save power by reducing the amount of data(e.g., pixel data) that needs to be read out from the AON camera sensor316. In some implementations, the AON camera processing subsystem 302can perform similar processing steps to the main camera processingsubsystem 304 on fewer pixels, resulting in fewer calculations andthereby reducing power consumption.

In some cases, a domain can include a set of image processing steps(e.g., noise reduction, edge enhancement, image stabilization, colorcorrection) performed on images or video frames captured by a camerasensor. In some implementations, video frames captured by the AON camerasensor 316 and the main camera sensor 318 can be processed withdifferent image processing steps. For example, the AON camera processingsubsystem 302 can perform fewer and/or different processing steps thanthe main camera processing subsystem 304. In some cases, performingfewer and/or different processing operations with the AON cameraprocessing subsystem 302 during AON operation can conserve power.

In another example, the AON camera sensor 316 can capture monochromevideo frames while main camera sensor 318 can capture red, green, blue(RGB) color video frames. In some cases, reading out and processingmonochrome video frames can consume less power than reading out andprocessing RGB color video frames. In some cases, the video framescaptured by the AON camera sensor 316 and the main camera sensor 318 canbe based on data captured from different portions of the light spectrumsuch as visible, ultraviolet (UV), near infrared (NIR), short waveinfrared (SWIR), other portions of the light spectrum, or anycombination thereof.

In the illustration of FIG. 3 , the AON camera sensor 316 and the maincamera sensor 318 can each provide frames to a different cameraprocessing subsystem. However, in some cases, a single camera processingsubsystem can process frames from the AON camera sensor 316 and the maincamera sensor 318 without departing from the scope of the presentdisclosure. Furthermore, although the AON camera sensor 316 and the maincamera sensor 318 are shown as two different sensors, one or more camerasensors that can operate in two or more different modes (e.g., an AONmode, a medium power mode, a high power mode, or the like) can also beused without departing from the scope of the present disclosure.

FIG. 4A and FIG. 4B illustrate example AON camera systemimplementations, according to some examples of the present disclosure.In one example illustrated in FIG. 4A, an AON camera sensor (e.g., AONcamera sensor 316 shown in FIG. 3 above) can be used to determinewhether an authorized user is interacting with a device (e.g., a mobilephone) in an AON face unlock implementation 402. In another exampleillustrated in FIG. 4A, an AON camera sensor can be used to providevision-based context sensing implementation 404. In the vision-basedcontext sensing implementation 404 illustrated in FIG. 4A, audio data,AON camera data, and/or data from one or more additional sensors can beused to determine whether a user is participating in a meeting and causea device to enter into a silent mode. In another illustrative example, avision-based context sensing implementation can include using acombination of inertial sensor data, audio sensor data, AON camera data,and/or data from one or more additional sensors to determine whether auser is speaking to a device (e.g., to give voice commands) or toanother person or device in the room. Another illustrative example of avision based context sensing implementation can be a combination ofinertial sensor data, audio sensor data, AON camera data, and/or datafrom one or more additional sensors camera to prevent device use by thedriver of a car. For example, the driver may be prevented from using amobile phone while a passenger may be permitted to use a mobile phone.In another example illustrated in FIG. 4A, an AON camera sensor can beused as part of an AON gesture detection implementation 406. Forexample, an AON camera sensor can be included in an XR system (e.g., anHMD), a mobile device, or any other device in an AON gesture detectionimplementation 406. In such an implementation, the AON camera cancapture images and perform gesture detection to allow a user to interactwith a device without making physical contact with the device.

FIG. 4B illustrates an example negative shutter lag video capturesequence 408. In the illustrative example of FIG. 4B, a user wearing anHMD or AR glasses is depicted. In the illustrated example, the user canbe observing 410 a scene. In some cases, while the user is observing 410the scene an event 412 can occur. In some cases, the event 412 can becaptured during AON operation (e.g., with AON camera sensor 316 shown inFIG. 3 above) included in the HMD. In the illustrated example, the usercan provide a capture input 414 to the HMD to initiate video capture. Insome cases, after the user initiates video capture, the HMD can begincapturing video frames in standard operation (e.g., with main camerasensor 318 shown in FIG. 3 above). As will be described in more detailwith respect to the figures below, the frames captured during AONoperation can be combined with the frames captured during standardoperation into a combined video that begins before the capture input. Insome cases, the frames captured during AON operation can be transformedto provide a consistent appearance of the combined video frames beforeand after the capture input. As noted above, references to AON operationherein can be understood to include capturing images and/or frames withone or more AON cameras and/or operating one or more cameras in an AONmode. Similarly, standard operation can be understood to includecapturing frames with one or more non-AON cameras and/or one or morecameras operating in a non-AON mode.

FIG. 5A illustrates an example block diagram of a negative shutter lagsystem 500, according to some examples of the disclosure. As shown, thecomponents of the negative shutter lag system 500 can include one ormore camera sensors 502, a camera pipeline 504, a domain switch 506, aframe buffer 512, a domain transform model 514, and a combiner 518. Insome cases, the one or more camera sensors 502 can include an AON camerasensor (e.g., AON camera sensor 316 shown in FIG. 3 ) and a main camerasensor (e.g., main camera sensor 318 shown in FIG. 3 ). In someexamples, the AON camera sensor can capture video frames 508 associatedwith a first domain and the main camera sensor can capture video frames510 associated with a second domain. In some cases, the one or morecamera sensors 502 can include one or more camera sensors that canoperate in two or more modes (e.g., an AON mode, a medium power more,and a high power mode). In some implementations, the one or more camerasensors 502 can capture video frames 508 associated with the firstdomain during AON operation and captured video frames 510 associatedwith the second domain during standard operation. In the illustratedexample, the time that the negative shutter lag system 500 receives acapture input 505 can be labeled as a time t=0. Video frames capturedbefore the capture input 505 occur during the range of time t<0 andvideo frames captured after the capture input 505 occur during the rangeof time t>0.

In some cases, frames captured by the one or more camera sensors 502 canbe processed by camera pipeline 504. For example, the camera pipeline504 can perform noise reduction, edge enhancement, image stabilization,color correction, and/or other image processing operations on the rawvideo frames provided by the one or more camera sensors 502.

In some examples, the domain switch 506 can be communicatively coupledwith the one or more camera sensors 502 and can control which camerasensor(s) and/or camera sensor mode(s) capture video frames. In somecases, the domain switch 506 can receive video frames associated withthe first domain before receiving capture input 505. For example, videoframes associated with the first domain can be received during AONoperation. In some cases, the domain switch 506 can route the videoframes 508 associated with the first domain to the frame buffer 512. Insome implementations, the domain switch 506 can enable adding additionalframes to the frame buffer 512 during AON operation and disable addingadditional frames to the frame buffer 512 during standard operation. Insome examples, frame buffer 512 can receive video frames 508 associatedwith the first domain directly from the one or more camera sensors 502and/or the camera pipeline 504.

In some cases, the negative shutter lag system 500 can receive a captureinput 505 (e.g., from a user pressing a button, performing a gesture, orthe like). Based on the capture input 505, the negative shutter lagsystem 500 can begin capturing video frames associated with the seconddomain. In some cases, the first video frame (or a subsequent videoframe) associated with the second domain captured after receiving thecapture input 505 can be stored or marked as a keyframe 511. In somecases, the keyframe 511 can be captured at approximately the same timeas receiving the capture input 505. After receiving the capture input505, the one or more camera sensors 502 and camera pipeline 504 canoutput video frames associated with the second domain until an endcapture input is received. In some cases, at the time of receiving thecapture input 505, the domain switch 506, one or more camera sensors502, and/or camera pipeline 504 can stop providing new frames to theframe buffer 512. In such cases, the video frames captured in the framebuffer 512 can span a period of time B based on the buffer length offrame buffer 512. In some cases, video frames stored in frame buffer 512can span the time period between t=−B and time the last video frameassociated with the first domain is captured before capture input 505t≈0.

In some cases, domain transform model 514 can be implemented as adeep-learning neural network. In some implementations, domain transformmodel 514 can be trained to transform video frames stored in framebuffer 512 associated with the first domain into transformed videoframes associated with the second domain. In some implementations, thedomain transform model 514 can be trained to use the keyframe 511associated with the second domain as a guide for transforming the videoframes associated with the first domain stored in the frame buffer 512.

In one illustrative example, a training dataset can include originalvideos and training videos. In some cases, all of the frames of theoriginal videos can be associated with the second domain. In some cases,a portion of each of the training videos (e.g., the first 30 frames,first 100 frames, first 300 frames, first 900 frames, or any othernumber of frames) can be associated with the first domain to simulatedata stored in the frame buffer 512 during AON operation. In some cases,the portion of the video frames associated with the first domain can begenerated from the original videos by transforming a portion of theoriginal video frames from the second domain to the first domain (e.g.,by downscaling resolution, converting from color to monochrome, reducingframerate, simulating different steps in the camera pipeline, or thelike). The original videos and the training videos can include akeyframe (e.g., keyframe 511) associated with the second domain that canbe used by the domain transform model 514 to transform the portion ofthe training videos associated with the first domain to the seconddomain.

In some cases, the resulting transformed video frames generated by thedomain transform model 514 can be directly compared to the originalvideo frames and a loss function can be used to determine an amount oferror between the transformed video frames and the original videoframes. The parameters (e.g., weights, biases, etc.) of the deeplearning network can be adjusted (or tuned) based on the error. Such atraining process can be referred to as supervised learning usingbackpropagation, which can be performed until the tuned parametersprovide a desired result.

In one example, the domain transform model 514 can be trained utilizinga deep generative neural network model (e.g., generative adversarialnetwork (GAN)). A GAN is a form of generative neural network that canlearn patterns in input data so that the neural network model cangenerate new synthetic outputs that reasonably could have been from theoriginal dataset. A GAN can include two neural networks that operatetogether. One of the neural networks (referred to as a generative neuralnetwork or generator denoted as G(z)) generates a synthesized output,and the other neural network (referred to as an discriminative neuralnetwork or discriminator denoted as D(X)) evaluates the synthesizedoutput for authenticity (whether the synthesized output is from anoriginal dataset, such as the training dataset, or is generated by thegenerator). The generator G(z) can correspond to the domain transformmodel 514. The generator is trained to try and fool the discriminatorinto determining a synthesized video frame (or group of video frames)generated by the generator is a real video frame (or group of videoframes) from a training dataset (e.g., the first group of training videodata). The training process continues, and the generator becomes betterat generating the synthetic video frames that look like real videoframes. The discriminator continues to find flaws in the synthesizedvideo frames, and the generator figures out what the discriminator islooking at to determine the flaws in the images. Once the network istrained, the generator is able to produce realistic looking video framesthat the discriminator is unable to distinguish from the real videoframes.

One example of a neural network that can be used for training of thedomain transform model 514 is a conditional GAN. In a conditional GAN,the generator is learning a conditional distribution. A conditional GANcan condition the generator and discriminator neural networks with somevector y, in which case the vector y is input into both the generatorand discriminator networks. Based on the vector y, the generator anddiscriminator become G(z,y) and D(X,y), respectively. The generatorG(z,y) models the distribution of the data, given z and y, in which casethe data X is generated as X˜G (X|z, y). The discriminator D(X,y)attempts to find a discriminating label for X and X_(G), which aremodeled with d˜D(d|X,y). The discriminator D(X, y) and the generatorG(z, y) are thus jointly conditioned to two variables z or X and y.

In one illustrative example, a conditional GAN can generate a videoframe (or a plurality of video frames) conditioned on a label (e.g.,represented as the vector y), where the label indicates the class of anobject that is in the video frame (or plurality of video frames), andthe goal of the GAN is transform a video frame (or plurality of videoframes) associated with the first domain to a video frame (or pluralityof video frames) associated with the second domain that includes thatclass of object. There is a dueling aspect between the generator G andthe discriminator D that is maximized with respect to the parameters ofthe discriminator D. The discriminator D will try to discriminatebetween real video frames (from the first group of training video data)and fake video frames (generated by the generator G based on the secondgroup of training video data) as well as possible, and the generator Gshould minimize the ability of the discriminator D to identify fakeimages. Parameters of the generator G (e.g., weights of the nodes of theneural network and in some cases other parameters, such as biases) canbe adjusted during the training process so that the generator G willoutput video frames that are indistinguishable from real video framesassociated with the second domain. A loss function can be used toanalyze errors in the generator G and discriminator D. In oneillustrative example, a binary cross-entropy loss function can be used.Other loss functions can be used in some cases.

During inference, (after the domain transform model 514 has beentrained), parameters of the domain transform model 514 (e.g., generatorG trained by the GAN) can be fixed and the domain transform model 514can transform video frames associated with the first domain (e.g., byupscaling resolution, colorizing, and/or performing any othertransformation) using the keyframe 511 as a guide to generatetransformed video frames 516 associated with the second domain.

In some cases, combiner 518 can combine the transformed video frames 516and the captured video frames 510 associated with the second domain fortime t>=0 into a combined video 520 having video frames associated withthe second domain. In one illustrative example, combiner 518 can performa concatenation of the transformed video frames 516 and the capturedvideo frames 510 associated with the second domain to generate thecombined video 520.

FIG. 5B illustrates a block diagram of another example negative shutterlag system 550. Similar to the negative shutter lag system 500 shown inFIG. 5A, the negative shutter lag system 550 can include one or morecamera sensors 502, frame buffer 512, domain transform model 514, andcombiner 518. In the illustrated example of FIG. 5B, the domain switch506 shown in FIG. 5A has been removed. In addition, instead of includinga single camera pipeline 504 as shown in FIG. 5A, negative shutter lagsystem 550 can include a first camera pipeline 522 and a second camerapipeline 524 communicatively coupled to the one or more camera sensors502.

As illustrated, the first camera pipeline 522 can receive video framesfrom the one or more camera sensors 502. In some cases, the first camerapipeline 522 can receive video frames from the one or more camerasensors 502 during AON operation of the one or more camera sensors 502.The first camera pipeline 522 can perform noise reduction, edgeenhancement, image stabilization, color correction, and/or other imageprocessing operations on the video frames from the one or more camerasensors 502 and the output of the first camera pipeline 522 can berouted to the frame buffer 512. The video frames output from the firstcamera pipeline 522 can be associated with a first domain. In somecases, the first domain can include the specific processing stepsperformed by the first camera pipeline 522. The output of the firstcamera pipeline 522 can include video frames 508 associated with thefirst domain.

The second camera pipeline 524 can also receive video frames from theone or more camera sensors 502. In some cases, the second camerapipeline 524 can receive video frames from the one or more camerasensors 502 during standard operation of the one or more camera sensors502. The second camera pipeline 524 can perform one or more imageprocessing operations on the received video frames. In some cases, theimage processing operations performed by the second camera pipeline 524can perform different processing steps and/or different numbers ofprocessing steps compared to the first camera pipeline 522. In somecases, the second domain can include the specific processing stepsperformed by the second camera pipeline 524. The output of the secondcamera pipeline 524 can include a keyframe 511 associated with thesecond domain and captured video frames 510 associated with the seconddomain. In some cases, when a capture input 505 is received (e.g., attime t=0), the one or more camera sensors 502 can pause outputting videoframes to the first camera pipeline 522 and begin outputting videoframes to the second camera pipeline 524.

As described above with respect to FIG. 5A, the domain transform model514 can be a deep learning neural network trained to transform videoframes from the first domain to the second domain. In the case of thenegative shutter lag system 550, generating the portion of each of thetraining videos associated with the first domain can include emulatingthe differences in processing steps between the first camera pipeline522 and the second camera pipeline 524. In some cases, emulating thedifferences in processing steps between the first camera pipeline 522and the second camera pipeline 524 can be done in addition toreplicating any other differences between the first domain and thesecond domain (e.g., resolution, color depth, framerate, or the like).Using the training set described above, the domain transform model 514can be trained in a supervised training process and/or trained with aGAN as described in the examples above, as well as any other suitabletraining technique.

During inference, (after the domain transform model 514 has beentrained), parameters of the domain transform model 514 (e.g., generatorG trained by the GAN) can be fixed and the domain transform model 514can transform video frames associated with the first domain (e.g., byupscaling resolution, colorizing, emulating image processing steps of acamera pipeline and/or performing any other transformation) using thekeyframe 511 as a guide to generate transformed video frames 516associated with the second domain.

FIG. 6 is a flow diagram illustrating an example of a process 600 ofprocessing one or more frames. At block 602, the process 600 includesobtaining a first plurality of frames associated with a first settingsdomain from an image capture system (e.g., image capture and processingsystem 100 shown in FIG. 1 , image sensor 202 shown in FIG. 2 , one ormore camera sensors 502) wherein the first plurality of frames iscaptured prior to obtaining a capture input. In some cases, the firstsettings domain includes a first resolution. In some cases, the firstsetting domain includes a first framerate.

At block 604, the process 600 includes obtaining at least one referenceframe (e.g., a keyframe) associated with a second settings domain fromthe image capture system. The at least one reference frame is capturedproximate to obtaining the capture input.

At block 606, the process 600 includes obtaining a second plurality offrames associated with the second settings domain from the image capturesystem. The second plurality of frames is captured after the at leastone reference frame. In some cases, the second settings domain includesa second resolution. In some cases, the second settings domain includesa second framerate.

At block 608, the process 600 includes based on the at least onereference frame, transforming (e.g., with domain transform model 514shown in FIG. 5A and FIG. 5B) at least a portion of the first pluralityof frames to generate a transformed plurality of frames associated withthe second settings domain. In some cases, transforming at least theportion of the first plurality of frames includes upscaling (e.g., withupscaling model 914 shown in FIG. 9 ) at least the portion of the firstplurality of frames from the first resolution to the second resolution.In some cases, transforming at least the portion of the first pluralityof frames includes converting (e.g., with framerate converter 1122 shownin FIG. 11 ) at least the portion of the first plurality of frames fromthe first framerate to the second framerate. While the followingexamples include discussion of the plurality of frames, the examples canbe associated with a portion of the plurality of frames (e.g., at leastthe portion of the first plurality of frames).

In some cases, process 600 includes combining the transformed pluralityof frames and the second plurality of frames to generate (e.g., withcombiner 1118) a video associated with the second settings domain.

In some cases, process 600 includes obtaining motion information (e.g.,from inertial motion estimator 1326 and/or optical motion estimator 1324shown in FIG. 13 ) associated with the first plurality of frames,wherein generating the transformed plurality of frames associated withthe second settings domain is based on the first plurality of frames,the at least one reference frame, and the motion information. In somecases, process 600 includes determining a panning direction based on themotion information associated with the first plurality of frames andapplying the panning direction to the transformed plurality of frames.

In some cases, process 600 includes capturing a first subset of thefirst plurality of frames at a first framerate and capturing a secondsubset of the first plurality of frames at a second framerate, differentfrom the first framerate. In some implementations, a change between thefirst framerate and the second framerate is based at least in part onmotion information associated with the first subset of the firstplurality of frames, the second subset of the first plurality of frames,or both.

In some cases, process 600 includes obtaining an additional referenceframe associated with the second settings domain from the image capturesystem. In some implementations, the additional reference frame iscaptured prior to obtaining the capture input. In some cases, generatingthe transformed plurality of frames associated with the second domain isbased on the first plurality of frames, the at least one referenceframe, and the additional reference frame. In some examples, the atleast one reference frame provides a reference for transforming at leasta first portion of the first plurality of frames and the additionalreference frame provides a reference for upscaling at least a secondportion of the first plurality of frames.

While the examples of the present disclosure describe various techniquesfor negative shutter lag related to video frames, the techniques of thepresent disclosure can also be utilized to provide similar negativeshutter lag for still images (also referred to as frames herein). Forexample, a user may have missed an interesting event and after the eventoccurs the user can initiate a capture input e.g., capture input 505shown in FIG. 5A and FIG. 5B) for a still image capture. In such cases,a negative shutter lag system such as negative shutter lag system 500shown in FIG. 5A or negative shutter lag system 550 shown in FIG. 5B canreceive a selection of one or more selected frames stored in the framebuffer 512 to be transformed from the first domain to the second domain.In some cases, the negative shutter lag system (or, e.g., a systemincluding the negative shutter lag system such as XR system 200) canpresent an interface for selecting the one or more selected frames. Forexample, the user can be provided with an interface for stepping throughand reviewing the video frames stored in frame buffer 512 frame-by-frameto select one or more selected frames for transformation. Otherillustrative examples of an interface for selecting the one or moreselected frames can include a slider for advancing through frames storedin the frame buffer 512 and a thumbnail gallery of the frames stored inthe frame buffer 512.

In some cases, the negative shutter lag system can suggest one or moreframes to transform. In one illustrative example, the negative shutterlag system can determine whether there has been a significant change ormotion that exceeds a threshold amount of change or motion in aparticular frame compared to other recent frames and suggest the frame(or frames) that exceed the threshold. For example, the negative shutterlag system may determine whether there has been significant change ormotion using a change detection algorithm to analyze changes or motionof a plurality of detected and/or tracked features. In anotherillustrative example, a deep learning neural network can be trained todetermine which frames contain relevant and/or interesting content basedon training with a dataset of labeled image data. In some cases, thedeep learning neural network can be trained on a personalized dataset ofimages that have been taken by a particular user. In anotherillustrative example, the negative shutter lag system can determine achange in the number of human faces as a basis for suggesting one ormore frames to transform. For example, a face detection neural networkcan be trained on a dataset of images of faces and non-faces todetermine the number of human faces in the frames stored in frame buffer512. Once the negative shutter lag system receives the selection of theone or more selected frames, domain transform model 514 can transformthe one or more selected frames to generate one or more transformedframes associated with the second domain.

FIG. 7 is a flow diagram illustrating an example of a process 700 ofprocessing one or more frames. At block 702, the process 700 includesobtaining a first plurality of frames associated with a first settingsdomain from an image capture system (e.g., image capture and processingsystem 100 shown in FIG. 1 , image sensor 202 shown in FIG. 2 , and/orone or more camera sensors 502), wherein the first plurality of framesis captured prior to obtaining a capture input.

At block 704, the process 700 includes obtaining a reference frame(e.g., a keyframe) associated with a second settings domain from theimage capture system, wherein the reference frame is captured proximateto obtaining the capture input.

At block 706, the process 700 includes obtaining a selection of one ormore selected frames associated with the first plurality of frames. Forexample, the user can be provided with an interface for stepping throughand reviewing the video frames stored in frame buffer (e.g., framebuffer 512 shown in FIG. 5A and FIG. 5B) frame-by-frame to select one ormore selected frames for transformation.

At block 708, the process 700 includes based on the reference frame,transforming (e.g., with domain transform model 514 shown in FIG. 5A andFIG. 5B) the one or more selected frames to generate one or moretransformed frames associated with the second settings domain.

FIG. 8A is a flow diagram illustrating an example process 800 forperforming a negative shutter lag capture. At block 802, the process 800can disable capture of video frames by a camera (e.g., AON camera sensor316 and/or main camera sensor 318 shown in FIG. 3 ).

At block 804, the process 800 can determine if AON recording is enabled.If the AON recording is disabled, the process 800 can return to block802. If AON recording is enabled, the process 800 can proceed to block806.

At block 806, the process 800 can capture frames associated with a firstdomain during AON operation. In one illustrative example, the firstdomain can include capturing monochrome frames from an NIR sensor. Insome cases, frames associated with the first domain can be stored in avideo buffer (e.g., SRAM 320, DRAM 314 shown in FIG. 3 , and/or storagedevice 1930, memory 1915, ROM 1920, RAM 1925 shown in FIG. 19 ). Forexample, the video buffer can include a circular buffer that storesvideo frames for a particular buffer length (e.g., 1 second, 5 seconds,30 seconds, 1 minute, 5 minutes, or any other amount of time). In somecases, the video buffer can accumulate video frames until the amount ofvideo stored in the video buffer spans the buffer length. Once the videobuffer is filled, each new frame captured at block 806 can replace theoldest frame held in the video buffer. As a result, the video buffer caninclude the most recent video frames captured at block 806 for a timespan based on the buffer length.

At block 808, the process 800 can determine whether a capture input hasbeen received. If the process 800 determines that a capture input hasnot been received, the process 800 can return to block 806 and continueAON operation. If the process 800 determines that the capture input hasbeen received, the process 800 can continue to block 810.

At block 810, the process 800 can capture video frames associated with asecond domain during standard operation. In one illustrative example,the second domain can include capturing RGB frames from a visible lightsensor. In some cases, at block 810, the contents of the video buffercan remain fixed during standard operation. In some cases, the videoframes associated with the second domain captured at block 810 can bestored in a separate memory or separate portion of memory from the videobuffer.

At block 812, the process 800 can determine if an end recording inputhas been received. If the end recording input was received, process 800can return to block 804. If the end recording input was not received,process 800 can return to block 810 and continue capturing frames duringstandard operation.

FIG. 8B illustrates a plot 850 of relative power consumption duringdifferent stages of the process 800 shown in FIG. 8A. Plot 850 is notshown to scale and is provided for illustrative purposes. In theillustrated example, the height of bar 822 indicates the relative powerconsumption of an image processing system (e.g., image processing system300 shown in FIG. 3 above) while capturing video frames associated witha first domain during AON operation (e.g., at block 806). In theillustrated example, the bar 822 can include power consumed by storingvideo frames captured during AON operation in a video buffer (e.g., acircular buffer). Similarly, in the illustrated example the height ofbar 824 can include the power consumed by one or more camera sensors(e.g., AON camera sensor 316 and/or main camera sensor 318 shown in FIG.3 above) during AON operation. In the illustrated example, the bufferlength 826 of the video buffer (e.g., DRAM 314 and/or SRAM 320 shown inFIG. 3 ) is illustrated by an arrow that ends at the time of the captureinput 827 and extends backward in time an amount of time based on thebuffer length B of the video buffer.

After the capture input 827 is received, the process 800 can capturevideo frames associated with a second domain during standard operation(e.g., at block 810). The height of bars 828 can represent the relativepower consumed by the image processing system during standard operationand shows an increase in power consumption relative to the powerconsumed during buffering video frames associated with the first domain.The increased power consumption by the image processing systemillustrated by bars 828 can be a result of, for example, processingvideo frames with larger numbers of pixels, more color information, ahigher framerate, more image processing steps, power differencesassociated with any other differences between the first domain and thesecond domain, or any combination thereof. Similarly, the bars 830 canrepresent the power of the one or more camera sensors during standardoperation. The increased power consumption by the one or more camerasensors illustrated by bars 830 can result from capturing andtransferring data for a larger number of pixels, with more colorinformation, at a higher framerate, power differences associated withany other differences between the first domain and the second domain, orany combination thereof.

In some cases, AON operation can continue for minutes, hours, or dayswithout the process 800 receiving a capture input. In such cases,reducing the power associated with capturing frames during AON operationwhen compared to standard operation can significantly increase theusable battery life of a negative shutter lag system. In some cases, anAON camera system that always captures and processes high power videoframes, both during standard operation and AON operation, can consumeavailable power (e.g., from a battery) much more quickly by comparison.

Video frame capture during standard operation can continue until process800 receives an end recording input (e.g., at block 812). As describedwith respect to negative shutter lag system 500 shown in FIG. 5A andnegative shutter lag system 550 shown in FIG. 5B, the video framesassociated with the first domain captured before the capture input 827and stored in the video buffer can be transformed based on the keyframeto generate transformed video frames associated with the second domain.In some cases, the transformed video frames associated with the seconddomain can be combined (e.g., at combiner 518) with the video framesassociated with the second domain to form a combined video. Squarebracket 832 illustrates the total time duration of an example combinedvideo based on the illustrated example of FIG. 8B.

FIG. 9 illustrates an example negative shutter lag system 900 accordingto examples of the present disclosure. As illustrated, the negativeshutter lag system 900 can include one or more camera sensors 902,camera pipeline 904, resolution switch 906, frame buffer 912, upscalingmodel 914, and combiner 918. In the example of FIG. 9 , frames 908captured before the negative shutter lag system 900 receives captureinput 905 can have a first resolution and frames 910 captured after thecapture input 905 is received can have a second resolution, differentfrom the first resolution. In one illustrative example, the firstresolution can be lower than the second resolution. Referring to FIG. 5Aand FIG. 5B, low resolution video frames 908 can be an illustrativeexample of frames associated with a first domain (e.g., video frames508) and the high resolution video frames 910 can be an illustrativeexample of video frames associated with a second domain (e.g., capturedvideo frames 510). The resolution switch 906 can be similar to andperform similar functions as the domain switch 506 shown in FIG. 5A. Forexample, the resolution switch 906 can be communicatively coupled withthe one or more camera sensors 902 and can control which camerasensor(s) and/or camera sensor mode(s) are used for capturing videoframes during AON operation and standard operation. In some cases,resolution switch 906 can receive video frames at the first resolutionfrom the one or more camera sensors 902 processed by the camera pipeline904 before capture input 905 is received. For example, low resolutionvideo frames can be received from an AON camera sensor and/or from acamera sensor operating in an AON mode. In some cases, resolution switch906 can route the low resolution video frames 908 to the frame buffer912. In some implementations, the resolution switch 906 can enable ordisable the frame buffer 912. In some cases, frame buffer 912 canreceive low resolution video frames 908 directly from the one or morecamera sensors 902 and/or the camera pipeline 904.

In some implementations, the frame buffer 912 can be similar to andperform similar functions as the frame buffer 512 shown in FIG. 5A. Forexample, the frame buffer 912 can have a buffer length B (e.g., onesecond, five seconds, ten seconds, thirty seconds, or any other selectedbuffer length). In some cases, after the capture input 905 is receivedby the negative shutter lag system 900, the video frames stored in framebuffer 912 can span the time period between t=−B and the time that thelast low resolution video frame is captured before capture input 905(e.g., t≈0)

In some cases, upscaling model 914 can be implemented as a deep-learningneural network. In some cases, upscaling model 914 can be trained toupscale the low resolution video frames 908 stored in frame buffer 912to the high resolution. In some implementations, the upscaling model 914can be trained to use the high resolution keyframe 911 as a guide forupscaling the low resolution video frames 908 stored in the frame buffer912.

In some cases, the upscaling model 914 can be trained using a processsimilar to the process described for training domain transform model 514described with respect to FIG. 5A. In the case of negative shutter lagsystem 900, a training dataset can include original videos and trainingvideos. In some cases, all of the frames of the original videos can behigh resolution frames. In some cases, a portion of each of the originalvideos (e.g., the first 30 frames, first 100 frames, first 300 frames,first 900 frames, or any other number of frames) can be downscaled tothe low resolution to simulate data stored in the frame buffer 512during AON operation to obtain the training videos. The original videosand the training videos can include a keyframe (e.g., keyframe 511) atthe high resolution that can be used by the domain transform model 514to upscale the low resolution portion of the training videos to the highresolution.

During inference, (after the upscaling model 914 has been trained),parameters of the upscaling model 914 (e.g., generator G trained withthe GAN) can be fixed and the upscaling model 914 can upscale lowresolution frames (e.g., frames stored in frame buffer 912) using thehigh resolution keyframe 911 as a guide to generate upscaled videoframes 916.

In some cases, combiner 918 can combine the upscaled video frames 916and the high resolution video frames 910 into a high resolution combinedvideo 920. In one illustrative example, combiner 518 can perform aconcatenation of the upscaled video frames 916 and the high resolutionvideo frames 910 to generate the combined video 920.

FIG. 10 is a flow diagram illustrating an example of a process 1000 forprocessing one or more video frames. At block 1002, the process 1000includes obtaining a first plurality of frames having a first resolutionfrom an image capture system (e.g., image capture and processing system100 shown in FIG. 1 , image sensor 202 shown in FIG. 2 , and/or one ormore camera sensors 502), wherein the first plurality of frames iscaptured prior to obtaining a capture input.

At block 1004, the process 1000 includes obtaining a reference frame(e.g., a keyframe) having a second resolution from the image capturesystem, wherein the reference frame is captured proximate to obtainingthe capture input.

At block 1006, the process 1000 includes obtaining a second plurality offrames having the second resolution from the image capture system,wherein the second plurality of frames is captured after the referenceframe.

At block 1008, the process 1000 includes based on the reference frame,upscaling (e.g., with domain transform model 514 shown in FIG. 5A andFIG. 5B and/or upscaling model 914 shown in FIG. 9 ) the first pluralityof frames from the first resolution to the second resolution to generatean upscaled plurality of frames having the second resolution.

FIG. 11 is a diagram illustrating another example negative shutter lagsystem 1100, in accordance with some examples. As illustrated, thenegative shutter lag system 1100 can include one or more camera sensors1102, camera pipeline 1104, domain switch 1106, frame buffer 1112,framerate converter 1122, upscaling model 1114, and combiner 1118. Oneor more components of the negative shutter lag system 1100 of FIG. 11can be similar to and perform similar operations as like numberedcomponents of the negative shutter lag system 900 of FIG. 9 . Forexample, the one or more camera sensors 1102 can be similar to andperform similar operations as the one or more camera sensors 902. Asanother example, camera pipeline 1104 can be similar to and performsimilar operations as camera pipeline 904. In the example of FIG. 11 ,frames 1108 captured before receiving the capture input 1105 can have afirst resolution and a first framerate. Video frames 1110 and keyframe1111 captured after receiving the capture input 1105 can have a secondresolution different from the first resolution and a second frameratedifferent from the second framerate. In one illustrative example, thesecond resolution can be higher than the first resolution and the secondframerate can be higher than the first framerate.

In order to generate upscaled and framerate adjusted video frames 1116that have the second resolution and the second framerate, the framerateconverter 1122 can adjust the framerate of frames 1108 from the firstframerate to the second framerate. In one illustrative example, theframerate converter 1122 can interpolate data from adjacent pairs offrames stored in the frame buffer 1112 to generate additional frames andthereby generate framerate adjusted frames 1125. In some cases, theupscaling model 1114, guided by the keyframe 1111, can upscale theresolution of the framerate adjusted frames 1125 to the secondresolution. The upscaling model 1114 can be similar to and performsimilar operations to the upscaling model 914 of FIG. 9 . The upscalingmodel 1114 can also be trained in a similar fashion to upscaling model914, as described in more detail with respect to FIG. 9 . In someimplementations, the resolution upscaling performed by upscaling model1114 and framerate adjustment performed by framerate converter 1122 canbe performed in the reverse order without departing from the scope ofthe present disclosure. In some cases, the upscaling model 1114, theframerate converter 1122, or a combination thereof can collectively beconsidered illustrative examples of a domain transform model (e.g.,corresponding to domain transform model 514 shown in FIG. 5A and FIG.5B).

In some cases, combiner 1118 can combine the upscaled and framerateadjusted video frames 1116 and the video frames 1110 with the secondresolution and second framerate into a combined video 1120 with thesecond resolution and second framerate. In one illustrative example,combiner 1118 can perform a concatenation of the upscaled and framerateadjusted video frames 1116 and the video frames 1110 with the secondresolution and second framerate to generate the combined video 1120.

FIG. 12 is a flow diagram illustrating an example of a process 1200 ofprocessing one or more frames. At block 1202, the process 1200 includesobtaining a first plurality of frames having a first resolution and afirst framerate from an image capture system (e.g., image capture andprocessing system 100 shown in FIG. 1 , image sensor 202 shown in FIG. 2, and/or one or more camera sensors 502), wherein the first plurality offrames is captured prior to obtaining a capture input.

At block 1204, the process 1200 includes obtaining a reference frame(e.g., a keyframe) having a second resolution from the image capturesystem, wherein the reference frame is captured proximate to obtainingthe capture input.

At block 1206, the process 1200 includes obtaining a second plurality offrames having the second resolution and a second framerate from theimage capture system, wherein the second plurality of frames is capturedafter the reference frame.

At block 1208, the process 1200 includes, based on the reference frame,upscaling the first plurality of frames from the first resolution to thesecond resolution and framerate adjusting the first plurality of framesfrom the first framerate to the second framerate to generate atransformed plurality of frames having the second resolution and thesecond framerate.

FIG. 13 is a diagram illustrating another example negative shutter lagsystem 1300, according to examples of the disclosure. As illustrated,the negative shutter lag system 1300 can include one or more camerasensors 1302, camera pipeline 1304, domain switch 1306, frame buffer1312, upscaling model 1314, combiner 1318, and framerate converter 1322.One or more components of the negative shutter lag system 1300 of FIG.13 can be similar to and perform similar operations as like componentsof the negative shutter lag system 1100 of FIG. 11 . For example, theone or more camera sensors 1302 can be similar to and perform similaroperations as the one or more camera sensors 1102. As another example,camera pipeline 1304 can be similar to and perform similar operations ascamera pipeline 1104.

As illustrated, negative shutter lag system 1300 can also includeframerate controller 1323, optical motion estimator 1324, and inertialmotion estimator 1326. In the negative shutter lag system 1300, frames1308 captured before receiving the capture input 1305 can be capturedwith a variable framerate. In some cases, the variable framerate of theframes 1308 can be determined by the framerate controller 1323.Framerate controller 1323 can receive inputs from one or more motionestimators. In the illustrative example shown in FIG. 13 , frameratecontroller 1323 can receive inputs from an inertial motion estimator1326 and an optical motion estimator 1324. In some cases, frameratecontroller 1323 can receive inputs from fewer (e.g., one), more (e.g.,three or more) and/or different types of motion estimators withoutdeparting from the scope of the present disclosure. In some cases,sensor data 1328 can be provided as an input to the inertial motionestimator 1326 from one or more sensors (e.g., accelerometer 204 and/orgyroscope 206 shown in FIG. 2 , an IMU, and/or any other motion sensor).In some cases, the inertial motion estimator 1326 can determine anestimated amount of motion of the one or more camera sensors 1302 basedon sensor data 1328. In some cases, optical motion estimator 1324 candetermine an amount of motion of the one or more camera sensors 1302and/or the scene being captured by the one or more camera sensors 1302based on the frames 1308 captured before receiving the capture input1305. In some cases, optical motion estimator can utilize one or moreoptical motion estimation techniques to estimate an amount of motion ofthe one or more camera sensors 1302 and/or the scene. In oneillustrative example, optical motion estimator 1324 can detect featuresin two or more of the frames 1308 (e.g., as part of 6DoF SLAM asdescribed with respect to FIG. 2 ). and determine an amount of motion ofone or more features between the two or more of the frames 1308.

In some cases, based on the motion estimates received from the inertialmotion estimator 1326 and/or the optical motion estimator 1324,framerate controller 1323 can vary the framerate of the frames 1308captured by the one or more camera sensors 1302 prior to receiving thecapture input 1305. For example, if the framerate controller 1323detects a relatively small amount of motion (or no motion) of the one ormore camera sensors 1302 and/or objects in the scene, the frameratecontroller 1323 can lower the framerate for frames 1308. In oneillustrative example, if the one or more camera sensors 1302 isstationary and capturing a static scene, the framerate controller 1323can lower the framerate. In some cases, the framerate controller 1323can lower the framerate of frames 1308 to one eighth, one quarter, onehalf, or any other fraction of the second framerate. In some cases, bylowering the framerate for frames 1308 before receiving the captureinput 1305 (e.g., frames captured during an AON mode), the powerconsumption can be reduced.

On the other hand, if the framerate controller 1323 detects a largeamount of motion from the inertial motion estimator 1326 and/or theoptical motion estimator 1324, the framerate controller 1323 canincrease the framerate. In one illustrative example, the frameratecontroller 1323 can increase the framerate if the one or more camerasensors 1302 are in motion and observing a sporting event (e.g., a scenecontaining a large amount of motion).

In some cases, the framerate controller 1323 can increase the framerateof the frames 1308 captured before receiving the capture input 1305 ashigh as the framerate used for capturing the frames 1310 after receivingthe capture input 1305. In some cases, the framerate controller 1323 canoutput the variable framerate associated with each of the frames 1308captured before receiving the capture input 1305 to the frame buffer1312.

In some cases, by increasing the framerate, the power consumption forcapturing the frames 1308 before receiving the capture input 1305 can beincreased. Although the power consumption can increase, by increasingthe framerate for frames 1308 during periods of increased motion, thequality of frames obtained during AON operation after framerateconversion and upscaling of the frames captured during AON operation canbe improved.

In some cases, the framerate converter 1322 can utilize the framerateinformation stored in the frame buffer 1312 to correctly performframerate conversion. For example, if the framerate controller 1323decreased the framerate of the frames 1308, the framerate converter 1322can generate additional frames (e.g., using an interpolation technique)to match the framerate of the frames 1310 captured after receiving thecapture input 1305.

In some cases, the framerate converter 1322 can adjust the framerate offrames 1308 from the variable framerate to the second framerate. In oneillustrative example, the framerate converter 1322 can interpolate datafrom adjacent pairs of frames stored in the frame buffer 1112 togenerate additional frames and thereby generate framerate adjustedframes 1325.

In on illustrative example, the frames 1308 captured before receivingthe capture input 1305 can have a variable framerate and a firstresolution. In addition, the frames 1310 and the keyframe 1311 capturedafter receiving the capture input 1305 can have a second resolution anda second framerate. In such an example, the upscaling model 1314, guidedby the keyframe 1311, can upscale the resolution of the framerateadjusted frames 1325 to the second resolution to generate framerateconverted and upscaled frames 1316. The upscaling model 1314 can besimilar to and perform similar operations to the upscaling model 914 ofFIG. 9 . The upscaling model 1314 can also be trained in a similarfashion to upscaling model 914, as described in more detail with respectto FIG. 9 . In some implementations, the resolution upscaling performedby upscaling model 1314 and framerate adjustment performed by framerateconverter 1322 can be performed in the reverse order without departingfrom the scope of the present disclosure. In some cases, the upscalingmodel 1314, the framerate converter 1322, or a combination thereof canbe considered illustrative examples of a domain transform model (e.g.,corresponding to domain transform model 514 shown in FIG. 5A and FIG.5B).

In some cases, inertial motion estimation data (e.g., from the inertialmotion estimator 1326) can be provided to the upscaling model 1314. Insome cases, the inertial motion estimation data can be stored in aninertial motion circular buffer (not shown). In some cases, the inertialestimation data can be associated with each video frame stored in theframe buffer 1312. In some cases, the upscaling model can be trained toutilize the inertial motion estimation data as well as the keyframe 1311as a guide for upscaling the framerate adjusted frames 1325 to thesecond resolution. For example, if the inertial motion estimation dataindicates that the one or more camera sensors 1302 were moving to theleft immediately preceding receiving the capture input 1305, theupscaling model 1314 can include a panning motion to the left in theconverted and upscaled frames 1316. In some cases, the upscaling model1314 can be trained with training data that includes simulated inertialmotion estimation data in a process similar to the training processdescribed for training upscaling model 914 with respect to FIG. 9 . Itshould be understood that the use of inertial motion estimation data asa guide can be utilized with any of the examples described herein. Forexample, inertial motion data can be used as a guide by domain transformmodel 514 shown in FIG. 5A and FIG. 5B.

FIG. 14 is a flow diagram illustrating an example of a process 1400 ofprocessing one or more frames. At block 1402, the process 1400 includescapturing video frames associated with a first domain during AONoperation. In some cases, the video frames captured during AON operationcan correspond to video frames 508 in FIG. 5A.

At block 1404, the process 1400 can determine if a capture input (e.g.,a user pressing a record button, performing a gesture, or the like) wasreceived. If a capture input was not received, the process 1400 canproceed to block 1406. At block 1406, the process 1400 can determinewhether capturing a keyframe associated with a second domain, differentfrom the first domain, is required. For example, the process 1400 maydetermine an amount of time since the most recent keyframe was acquired.In some cases, the period between successively captured keyframes canhave a fixed value. In some cases, the period between successivelycaptured keyframes can be variable. In one illustrative example, theperiod between successively captured keyframes can be determined basedon motion estimation from one or more motion estimators (e.g., inertialmotion estimator 1326, optical motion estimator 1324, any other motionestimator, or a combination thereof).

If the capture input has been received at block 1404, the process 1400can proceed to block 1410. At block 1410, the process 1400 can capture akeyframe (e.g., keyframe 511 shown in FIG. 5A) associated with thesecond domain at approximately the time of the capture input (e.g.,capture input 505 shown in FIG. 5A).

At block 1412, the process 1400 can capture video frames associated withthe second domain (e.g., captured video frames 510 associated with thesecond domain shown in FIG. 5A) during standard operation.

At block 1414, the process 1400 can determine if an end recording inputwas received. If the end recording input was not received, the process1400 can continue capturing video frames associated with the seconddomain during standard operation at block 1412. If the end recordinginput was received, the process 1400 can return to block 1402 if AONoperation is enabled.

In some cases, after receiving the end recording input at block 1414,the process 1400 can include transforming the video frames associatedwith the first domain captured at block 1402 to the second domain with adomain transform model (e.g., domain transform model 514 shown in FIG.5A). In some cases, each of the keyframes captured at block 1408 can beused as a guide for transforming a portion of the video framesassociated with the first domain. In some cases, each of the keyframescaptured at block 1408 can be used as a guide for transforming framesfrom the first plurality of frames that are temporally local to each ofthe keyframes captured at block 1408. In one implementation, aparticular keyframe captured at block 1408 can be used as a guide totransform a subset of the video frames associated with the first domaincaptured at block 1402 that were captured before the particular keyframeand were also captured after the keyframe immediately preceding theparticular keyframe. In one illustrative example, the particularkeyframe used to transform each video frame associated with the firstdomain captured at block 1402 can be the keyframe captured closest intime to each respective video frame. In another illustrative example,selecting which of the keyframes to use as a guide for a particularsubset of the video frames associated with the first domain captured atblock 1402 can be based at least in part on performing feature detectionin the keyframe and one or more of the video frames associated with thefirst domain and comparing the detected features to determine the bestkeyframe (or keyframes) to utilize as a guide. For example, a featurematching technique can be used to determine which keyframe chosen frombetween the two keyframes nearest in time to each respective video framebeing transformed at block 1414 is closest in content. In such anexample, the keyframe closest in content to the respective video framecan be used as the guide.

FIG. 15 illustrates a plot 1550 of relative power consumption duringdifferent stages of process 1400 shown in FIG. 14 . Plot 1550 is notshown to scale and is provided for illustration purposes. In theillustrated example, the height of bar 1522 indicates the relative powerconsumption of an image processing system (e.g., image processing system300 shown in FIG. 3 above) while the process 1400 captures video framesassociated with the first domain during AON operation (e.g., at block1402). In the illustrated example, the bar 1522 can include powerconsumption of storing video frames associated with the first domain ina video buffer (e.g., a circular buffer). Similarly, in the illustratedexample the height of bar 1524 can indicate the power consumed by theone or more camera sensors while capturing frames associated with thefirst domain (e.g., AON camera sensor 316 and/or main camera sensor 318shown in FIG. 3 above). In the illustrated example of FIG. 15 , afterreceiving the capture input 1527 (e.g., at block 1408), the process 1400can capture video frames associated with the second domain (e.g., atblock 1410). In the illustrated example, the buffer length 1526 of thevideo buffer (e.g., DRAM 314, SRAM 320 shown in FIG. 3 ) is illustratedby an arrow that ends at the time of the capture input 1527 and extendsbackward in time an amount of time based on the buffer length B of thevideo buffer.

After receiving the capture input 1527, the process 1400 can capturevideo frames associated with the second domain (e.g., at block 1410).The bars 1528 can represent the power consumed by the image processingsystem during capture of the video frames associated with the seconddomain and shows an increase in power consumption relative to the powerconsumed during buffering video frames associated with the first domain.The increased power consumption by the image processing system can be aresult of, for example, processing video frames with larger numbers ofpixels, more color information, a higher framerate, more and/ordifferent image processing steps, power differences associated with anyother differences between the first domain and the second domain, or anycombination thereof. The bars 1529 can represent the power consumed bythe image processing system during capture of keyframes associated withthe second domain prior to receiving the capture input 1527 (e.g., atblock 1406). As illustrated, capturing keyframes associated with thesecond domain can consume comparable amounts of power to capturing videoframes associated with the second domain after receiving the captureinput 1527.

The bars 1530 can represent the power of the one or more camera sensorswhile capturing video frames associated with the second domain. Theincreased power consumption by the one or more camera sensors can resultfrom capturing and transferring data for a larger number of pixels, withmore color information, at a higher framerate, power differencesassociated with any other differences between the first domain and thesecond domain, or any combination thereof. The bars 1531 can representpower consumption by the one or more camera sensors during capture ofkeyframes associated with the second domain prior to receiving thecapture input 1527 (e.g., at block 1406).

Prior to receiving the capture input 1527, the period 1534 betweensuccessive keyframes can be determined at least in part on estimatedmotion of the one or more camera sensors and/or the scene captured bythe one or more camera sensors.

Capture of video frames associated with the second domain can continueuntil process 1400 receives an end recording input (e.g., at block1412). In some cases, the portion of the video frames associated withthe first domain captured before the capture input 1527 stored in thevideo buffer can be transformed from the first domain to the seconddomain by a domain transform model (e.g., domain transform model 514shown in FIG. 5A and FIG. 5B) to generate transformed video framesassociated with the second domain. The transformed video frames can becombined (e.g., by combiner 518 shown in FIG. 5A and FIG. 5B) with thevideo frames associated with the second domain captured after thecapture input 1527 to form a combined video. Square bracket 1532illustrates the total time duration of an example combined video basedon the illustrated example of FIG. 15 .

FIG. 16 is a flow diagram illustrating an example of a process 1600 ofprocessing one or more frames. At block 1602, the process 1600 includesobtaining a first plurality of frames associated with a first settingsdomain from an image capture system (e.g., image capture and processingsystem 100 shown in FIG. 1 , image sensor 202 shown in FIG. 2 , and/orone or more camera sensors 502), wherein the first plurality of framesis captured prior to obtaining a capture input.

At block 1604, the process 1600 includes obtaining a first referenceframe (e.g., a keyframe) associated with a second settings domain fromthe image capture system, wherein the first reference frame is capturedprior to obtaining the capture input.

At block 1606, the process 1600 includes obtaining a second referenceframe associated with the second settings domain from the image capturesystem, wherein the second reference frame is captured proximate toobtaining the capture input.

At block 1608, the process 1600 includes obtaining a second plurality offrames associated with the second settings domain from the image capturesystem, wherein the second plurality of frames is captured after thesecond reference frame.

At block 1610, the process 1600 includes, based on the first referenceframe, transforming at least a portion of the first plurality of framesto generate a first transformed plurality of frames associated with thesecond settings domain.

At block 1612, the process 1600 includes, based on the second referenceframe, transforming at least another portion of the first plurality offrames to generate a second transformed plurality of frames associatedwith the second settings domain.

In some cases, process 1600 includes obtaining a motion estimateassociated with the first plurality of frames and obtaining a thirdreference frame associated with the second settings domain from theimage capture system, wherein the third reference frame is capturedprior to obtaining the capture input. In some cases, an amount of timebetween the first reference frame and the third reference frame is basedon the motion estimate associated with the first plurality of frames.

In some examples, the processes described herein (e.g., processes 600,700, 800, 1000, 1200, 1400, 1600 and/or other process described herein)may be performed by a computing device or apparatus. In one example, oneor more of the processes can be performed by the image processing system300 of FIG. 3 . In another example, one or more of the processes can beperformed by the computing system 1900 shown in FIG. 19 . For instance,a computing device with the computing system 1900 shown in FIG. 19 caninclude the components of the negative shutter lag system 500, negativeshutter lag system 550, negative shutter lag system 900, negativeshutter lag system 1100, negative shutter lag system 1300, or anycombination thereof and can implement the operations of the process 600of FIG. 6 , process 700 of FIG. 7 , process 800 of FIG. 8A, process 1000of FIG. 10 , process 1200 of FIG. 12 , process 1400 of FIG. 14 , process1600 of FIG. 16 and/or other processes described herein.

The computing device can include any suitable device, such as a vehicleor a computing device of a vehicle (e.g., a driver monitoring system(DMS) of a vehicle), a mobile device (e.g., a mobile phone), a desktopcomputing device, a tablet computing device, a wearable device (e.g., aVR headset, an AR headset, AR glasses, a network-connected watch orsmartwatch, or other wearable device), a server computer, a roboticdevice, a television, and/or any other computing device with theresource capabilities to perform the processes described herein,including the processes 600, 700, 800, 1000, 1200, 1400, 1600 and/orother process described herein. In some cases, the computing device orapparatus may include various components, such as one or more inputdevices, one or more output devices, one or more processors, one or moremicroprocessors, one or more microcomputers, one or more cameras, one ormore sensors, and/or other component(s) that are configured to carry outthe steps of processes described herein. In some examples, the computingdevice may include a display, a network interface configured tocommunicate and/or receive the data, any combination thereof, and/orother component(s). The network interface may be configured tocommunicate and/or receive Internet Protocol (IP) based data or othertype of data.

The components of the computing device can be implemented in circuitry.For example, the components can include and/or can be implemented usingelectronic circuits or other electronic hardware, which can include oneor more programmable electronic circuits (e.g., microprocessors,graphics processing units (GPUs), digital signal processors (DSPs),central processing units (CPUs), and/or other suitable electroniccircuits), and/or can include and/or be implemented using computersoftware, firmware, or any combination thereof, to perform the variousoperations described herein.

The processes 600, 700, 800, 1000, 1200, 1400, and 1600 are illustratedas logical flow diagrams, the operation of which represents a sequenceof operations that can be implemented in hardware, computerinstructions, or a combination thereof. In the context of computerinstructions, the operations represent computer-executable instructionsstored on one or more computer-readable storage media that, whenexecuted by one or more processors, perform the recited operations.Generally, computer-executable instructions include routines, programs,objects, components, data structures, and the like that performparticular functions or implement particular data types. The order inwhich the operations are described is not intended to be construed as alimitation, and any number of the described operations can be combinedin any order and/or in parallel to implement the processes.

Additionally, the processes 600, 700, 800, 1000, 1200, 1400, 1600 and/orother process described herein may be performed under the control of oneor more computer systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs, or one or more applications) executing collectivelyon one or more processors, by hardware, or combinations thereof. Asnoted above, the code may be stored on a computer-readable ormachine-readable storage medium, for example, in the form of a computerprogram comprising a plurality of instructions executable by one or moreprocessors. The computer-readable or machine-readable storage medium maybe non-transitory.

As noted above, various aspects of the present disclosure can usemachine learning models or systems. FIG. 17 is an illustrative exampleof a deep learning neural network 1700 that can be used to implementmachine learning based feature extraction and/or activity recognition(or classification). In one illustrative example, feature extractionand/or activity recognition can be used by a discriminator networkduring training of a guided domain transform model (e.g., as describedwith respect to FIG. 5A) and/or guided super-resolution model (e.g., asdescribed with respect to FIG. 9 ) utilizing a GAN as described above.An input layer 1720 includes input data. In one illustrative example,the input layer 1720 can include data representing the pixels of aninput video frame. The neural network 1700 includes multiple hiddenlayers 1722 a, 1722 b, through 1722 n. The hidden layers 1722 a, 1722 b,through 1722 n include “n” number of hidden layers, where “n” is aninteger greater than or equal to one. The number of hidden layers can bemade to include as many layers as needed for the given application. Theneural network 1700 further includes an output layer 1721 that providesan output resulting from the processing performed by the hidden layers1722 a, 1722 b, through 1722 n. In one illustrative example, the outputlayer 1721 can provide a classification for an object in an input videoframe. The classification can include a class identifying the type ofactivity (e.g., looking up, looking down, closing eyes, yawning, etc.).

The neural network 1700 is a multi-layer neural network ofinterconnected nodes. Each node can represent a piece of information.Information associated with the nodes is shared among the differentlayers and each layer retains information as information is processed.In some cases, the neural network 1700 can include a feed-forwardnetwork, in which case there are no feedback connections where outputsof the network are fed back into itself. In some cases, the neuralnetwork 1700 can include a recurrent neural network, which can haveloops that allow information to be carried across nodes while reading ininput.

Information can be exchanged between nodes through node-to-nodeinterconnections between the various layers. Nodes of the input layer1720 can activate a set of nodes in the first hidden layer 1722 a. Forexample, as shown, each of the input nodes of the input layer 1720 isconnected to each of the nodes of the first hidden layer 1722 a. Thenodes of the first hidden layer 1722 a can transform the information ofeach input node by applying activation functions to the input nodeinformation. The information derived from the transformation can then bepassed to and can activate the nodes of the next hidden layer 1722 b,which can perform their own designated functions. Example functionsinclude convolutional, up-sampling, data transformation, and/or anyother suitable functions. The output of the hidden layer 1722 b can thenactivate nodes of the next hidden layer, and so on. The output of thelast hidden layer 1722 n can activate one or more nodes of the outputlayer 1721, at which an output is provided. In some cases, while nodes(e.g., node 1726) in the neural network 1700 are shown as havingmultiple output lines, a node has a single output and all lines shown asbeing output from a node represent the same output value.

In some cases, each node or interconnection between nodes can have aweight that is a set of parameters derived from the training of theneural network 1700. Once the neural network 1700 is trained, it can bereferred to as a trained neural network, which can be used to classifyone or more activities. For example, an interconnection between nodescan represent a piece of information learned about the interconnectednodes. The interconnection can have a tunable numeric weight that can betuned (e.g., based on a training dataset), allowing the neural network1700 to be adaptive to inputs and able to learn as more and more data isprocessed.

The neural network 1700 is pre-trained to process the features from thedata in the input layer 1720 using the different hidden layers 1722 a,1722 b, through 1722 n in order to provide the output through the outputlayer 1721. In an example in which the neural network 1700 is used toidentify activities being performed by a driver in frames, the neuralnetwork 1700 can be trained using training data that includes bothframes and labels, as described above. For instance, training frames canbe input into the network, with each training frame having a labelindicating the features in the frames (for the feature extractionmachine learning system) or a label indicating classes of an activity ineach frame. In one example using object classification for illustrativepurposes, a training frame can include an image of a number 2, in whichcase the label for the image can be [0 0 1 0 0 0 0 0 0 0].

In some cases, the neural network 1700 can adjust the weights of thenodes using a training process called backpropagation. As noted above, abackpropagation process can include a forward pass, a loss function, abackward pass, and a weight update. The forward pass, loss function,backward pass, and parameter update is performed for one trainingiteration. The process can be repeated for a certain number ofiterations for each set of training images until the neural network 1700is trained well enough so that the weights of the layers are accuratelytuned.

For the example of identifying objects in frames, the forward pass caninclude passing a training frame through the neural network 1700. Theweights are initially randomized before the neural network 1700 istrained. As an illustrative example, a frame can include an array ofnumbers representing the pixels of the image. Each number in the arraycan include a value from 0 to 255 describing the pixel intensity at thatposition in the array. In one example, the array can include a 28×28×3array of numbers with 28 rows and 28 columns of pixels and 3 colorcomponents (such as red, green, and blue, or luma and two chromacomponents, or the like).

As noted above, for a first training iteration for the neural network1700, the output will likely include values that do not give preferenceto any particular class due to the weights being randomly selected atinitialization. For example, if the output is a vector withprobabilities that the object includes different classes, theprobability value for each of the different classes may be equal or atleast very similar (e.g., for ten possible classes, each class may havea probability value of 0.1). With the initial weights, the neuralnetwork 1700 is unable to determine low level features and thus cannotmake an accurate determination of what the classification of the objectmight be. A loss function can be used to analyze error in the output.Any suitable loss function definition can be used, such as aCross-Entropy loss. Another example of a loss function includes the meansquared error (MSE), defined as

$E_{total} = {\Sigma\frac{1}{2}{( {{target} - {output}} )^{2}.}}$

The loss can be set to be equal to the value of E_(total).

The loss (or error) will be high for the first training images since theactual values will be much different than the predicted output. The goalof training is to minimize the amount of loss so that the predictedoutput is the same as the training label. The neural network 1700 canperform a backward pass by determining which inputs (weights) mostcontributed to the loss of the network, and can adjust the weights sothat the loss decreases and is eventually minimized. A derivative of theloss with respect to the weights (denoted as dL/dW, where W are theweights at a particular layer) can be computed to determine the weightsthat contributed most to the loss of the network. After the derivativeis computed, a weight update can be performed by updating all theweights of the filters. For example, the weights can be updated so thatthey change in the opposite direction of the gradient. The weight updatecan be denoted as

${w = {w_{i} - {\eta\frac{dL}{dW}}}},$

where w denotes a weight, w_(i) denotes the initial weight, and fdenotes a learning rate. The learning rate can be set to any suitablevalue, with a high learning rate including larger weight updates and alower value indicating smaller weight updates.

The neural network 1700 can include any suitable deep network. Oneexample includes a convolutional neural network (CNN), which includes aninput layer and an output layer, with multiple hidden layers between theinput and out layers. The hidden layers of a CNN include a series ofconvolutional, nonlinear, pooling (for downsampling), and fullyconnected layers. The neural network 1700 can include any other deepnetwork other than a CNN, such as an autoencoder, a deep belief nets(DBNs), a Recurrent Neural Networks (RNNs), among others.

FIG. 18 is an illustrative example of a convolutional neural network(CNN) 1800. The input layer 1820 of the CNN 1800 includes datarepresenting an image or frame. For example, the data can include anarray of numbers representing the pixels of the image, with each numberin the array including a value from 0 to 255 describing the pixelintensity at that position in the array. Using the previous example fromabove, the array can include a 28×28×3 array of numbers with 28 rows and28 columns of pixels and 3 color components (e.g., red, green, and blue,or luma and two chroma components, or the like). The image can be passedthrough a convolutional hidden layer 1822 a, an optional non-linearactivation layer, a pooling hidden layer 1822 b, and fully connectedhidden layers 1822 c to get an output at the output layer 1824. Whileonly one of each hidden layer is shown in FIG. 18 , one of ordinaryskill will appreciate that multiple convolutional hidden layers,non-linear layers, pooling hidden layers, and/or fully connected layerscan be included in the CNN 1800. As previously described, the output canindicate a single class of an object or can include a probability ofclasses that best describe the object in the image.

The first layer of the CNN 1800 is the convolutional hidden layer 1822a. The convolutional hidden layer 1822 a analyzes the image data of theinput layer 1820. Each node of the convolutional hidden layer 1822 a isconnected to a region of nodes (pixels) of the input image called areceptive field. The convolutional hidden layer 1822 a can be consideredas one or more filters (each filter corresponding to a differentactivation or feature map), with each convolutional iteration of afilter being a node or neuron of the convolutional hidden layer 1822 a.For example, the region of the input image that a filter covers at eachconvolutional iteration would be the receptive field for the filter. Inone illustrative example, if the input image includes a 28×28 array, andeach filter (and corresponding receptive field) is a 5×5 array, thenthere will be 24×24 nodes in the convolutional hidden layer 1822 a. Eachconnection between a node and a receptive field for that node learns aweight and, in some cases, an overall bias such that each node learns toanalyze its particular local receptive field in the input image. Eachnode of the hidden layer 1822 a will have the same weights and bias(called a shared weight and a shared bias). For example, the filter hasan array of weights (numbers) and the same depth as the input. A filterwill have a depth of 3 for the video frame example (according to threecolor components of the input image). An illustrative example size ofthe filter array is 5×5×3, corresponding to a size of the receptivefield of a node.

The convolutional nature of the convolutional hidden layer 1822 a is dueto each node of the convolutional layer being applied to itscorresponding receptive field. For example, a filter of theconvolutional hidden layer 1822 a can begin in the top-left corner ofthe input image array and can convolve around the input image. As notedabove, each convolutional iteration of the filter can be considered anode or neuron of the convolutional hidden layer 1822 a. At eachconvolutional iteration, the values of the filter are multiplied with acorresponding number of the original pixel values of the image (e.g.,the 5×5 filter array is multiplied by a 5×5 array of input pixel valuesat the top-left corner of the input image array). The multiplicationsfrom each convolutional iteration can be summed together to obtain atotal sum for that iteration or node. The process is next continued at anext location in the input image according to the receptive field of anext node in the convolutional hidden layer 1822 a. For example, afilter can be moved by a step amount (referred to as a stride) to thenext receptive field. The stride can be set to 1 or other suitableamount. For example, if the stride is set to 1, the filter will be movedto the right by 1 pixel at each convolutional iteration. Processing thefilter at each unique location of the input volume produces a numberrepresenting the filter results for that location, resulting in a totalsum value being determined for each node of the convolutional hiddenlayer 1822 a.

The mapping from the input layer to the convolutional hidden layer 1822a is referred to as an activation map (or feature map). The activationmap includes a value for each node representing the filter results ateach locations of the input volume. The activation map can include anarray that includes the various total sum values resulting from eachiteration of the filter on the input volume. For example, the activationmap will include a 24×24 array if a 5×5 filter is applied to each pixel(a stride of 1) of a 28×28 input image. The convolutional hidden layer1822 a can include several activation maps in order to identify multiplefeatures in an image. The example shown in FIG. 18 includes threeactivation maps. Using three activation maps, the convolutional hiddenlayer 1822 a can detect three different kinds of features, with eachfeature being detectable across the entire image.

In some examples, a non-linear hidden layer can be applied after theconvolutional hidden layer 1822 a. The non-linear layer can be used tointroduce non-linearity to a system that has been computing linearoperations. One illustrative example of a non-linear layer is arectified linear unit (ReLU) layer. A ReLU layer can apply the functionf(x)=max(0, x) to all of the values in the input volume, which changesall the negative activations to 0. The ReLU can thus increase thenon-linear properties of the CNN 1800 without affecting the receptivefields of the convolutional hidden layer 1822 a.

The pooling hidden layer 1822 b can be applied after the convolutionalhidden layer 1822 a (and after the non-linear hidden layer when used).The pooling hidden layer 1822 b is used to simplify the information inthe output from the convolutional hidden layer 1822 a. For example, thepooling hidden layer 1822 b can take each activation map output from theconvolutional hidden layer 1822 a and generates a condensed activationmap (or feature map) using a pooling function. Max-pooling is oneexample of a function performed by a pooling hidden layer. Other formsof pooling functions be used by the pooling hidden layer 1822 a, such asaverage pooling, L2-norm pooling, or other suitable pooling functions. Apooling function (e.g., a max-pooling filter, an L2-norm filter, orother suitable pooling filter) is applied to each activation mapincluded in the convolutional hidden layer 1822 a. In the example shownin FIG. 18 , three pooling filters are used for the three activationmaps in the convolutional hidden layer 1822 a.

In some examples, max-pooling can be used by applying a max-poolingfilter (e.g., having a size of 2×2) with a stride (e.g., equal to adimension of the filter, such as a stride of 2) to an activation mapoutput from the convolutional hidden layer 1822 a. The output from amax-pooling filter includes the maximum number in every sub-region thatthe filter convolves around. Using a 2×2 filter as an example, each unitin the pooling layer can summarize a region of 2×2 nodes in the previouslayer (with each node being a value in the activation map). For example,four values (nodes) in an activation map will be analyzed by a 2×2max-pooling filter at each iteration of the filter, with the maximumvalue from the four values being output as the “max” value. If such amax-pooling filter is applied to an activation filter from theconvolutional hidden layer 1822 a having a dimension of 24×24 nodes, theoutput from the pooling hidden layer 1822 b will be an array of 12×12nodes.

In some examples, an L2-norm pooling filter could also be used. TheL2-norm pooling filter includes computing the square root of the sum ofthe squares of the values in the 2×2 region (or other suitable region)of an activation map (instead of computing the maximum values as is donein max-pooling), and using the computed values as an output.

Intuitively, the pooling function (e.g., max-pooling, L2-norm pooling,or other pooling function) determines whether a given feature is foundanywhere in a region of the image, and discards the exact positionalinformation. This can be done without affecting results of the featuredetection because, once a feature has been found, the exact location ofthe feature is not as important as its approximate location relative toother features. Max-pooling (as well as other pooling methods) offer thebenefit that there are many fewer pooled features, thus reducing thenumber of parameters needed in later layers of the CNN 1800.

The final layer of connections in the network is a fully-connected layerthat connects every node from the pooling hidden layer 1822 b to everyone of the output nodes in the output layer 1824. Using the exampleabove, the input layer includes 28×28 nodes encoding the pixelintensities of the input image, the convolutional hidden layer 1822 aincludes 3×24×24 hidden feature nodes based on application of a 5×5local receptive field (for the filters) to three activation maps, andthe pooling hidden layer 1822 b includes a layer of 3×12×12 hiddenfeature nodes based on application of max-pooling filter to 2×2 regionsacross each of the three feature maps. Extending this example, theoutput layer 1824 can include ten output nodes. In such an example,every node of the 3×12×12 pooling hidden layer 1822 b is connected toevery node of the output layer 1824.

The fully connected layer 1822 c can obtain the output of the previouspooling hidden layer 1822 b (which should represent the activation mapsof high-level features) and determines the features that most correlateto a particular class. For example, the fully connected layer 1822 clayer can determine the high-level features that most strongly correlateto a particular class, and can include weights (nodes) for thehigh-level features. A product can be computed between the weights ofthe fully connected layer 1822 c and the pooling hidden layer 1822 b toobtain probabilities for the different classes. For example, if the CNN1800 is being used to predict that an object in a video frame is aperson, high values will be present in the activation maps thatrepresent high-level features of people (e.g., two legs are present, aface is present at the top of the object, two eyes are present at thetop left and top right of the face, a nose is present in the middle ofthe face, a mouth is present at the bottom of the face, and/or otherfeatures common for a person).

In some examples, the output from the output layer 1824 can include anM-dimensional vector (in the prior example, M=10). M indicates thenumber of classes that the CNN 1800 has to choose from when classifyingthe object in the image. Other example outputs can also be provided.Each number in the M-dimensional vector can represent the probabilitythe object is of a certain class. In one illustrative example, if a10-dimensional output vector represents ten different classes of objectsis [0 0 0.05 0.8 0 0.15 0 0 0 0], the vector indicates that there is a5% probability that the image is the third class of object (e.g., adog), an 80% probability that the image is the fourth class of object(e.g., a human), and a 15% probability that the image is the sixth classof object (e.g., a kangaroo). The probability for a class can beconsidered a confidence level that the object is part of that class.

FIG. 19 is a diagram illustrating an example of a system forimplementing certain aspects of the present technology. In particular,FIG. 19 illustrates an example of computing system 1900, which can befor example any computing device making up internal computing system, aremote computing system, a camera, or any component thereof in which thecomponents of the system are in communication with each other usingconnection 1905. Connection 1905 can be a physical connection using abus, or a direct connection into processor 1910, such as in a chipsetarchitecture. Connection 1905 can also be a virtual connection,networked connection, or logical connection.

In some embodiments, computing system 1900 is a distributed system inwhich the functions described in this disclosure can be distributedwithin a datacenter, multiple data centers, a peer network, etc. In someembodiments, one or more of the described system components representsmany such components each performing some or all of the function forwhich the component is described. In some embodiments, the componentscan be physical or virtual devices.

Example computing system 1900 includes at least one processing unit (CPUor processor) 1910 and connection 1905 that couples various systemcomponents including system memory 1915, such as read-only memory (ROM)1920 and random access memory (RAM) 1925 to processor 1910. Computingsystem 1900 can include a cache 1912 of high-speed memory connecteddirectly with, in close proximity to, or integrated as part of processor1910.

Processor 1910 can include any general purpose processor and a hardwareservice or software service, such as services 1932, 1934, and 1936stored in storage device 1930, configured to control processor 1910 aswell as a special-purpose processor where software instructions areincorporated into the actual processor design. Processor 1910 mayessentially be a completely self-contained computing system, containingmultiple cores or processors, a bus, memory controller, cache, etc. Amulti-core processor may be symmetric or asymmetric.

To enable user interaction, computing system 1900 includes an inputdevice 1945, which can represent any number of input mechanisms, such asa microphone for speech, a touch-sensitive screen for gesture orgraphical input, keyboard, mouse, motion input, speech, etc. Computingsystem 1900 can also include output device 1935, which can be one ormore of a number of output mechanisms. In some instances, multimodalsystems can enable a user to provide multiple types of input/output tocommunicate with computing system 1900. Computing system 1900 caninclude communications interface 1940, which can generally govern andmanage the user input and system output. The communication interface mayperform or facilitate receipt and/or transmission wired or wirelesscommunications using wired and/or wireless transceivers, including thosemaking use of an audio jack/plug, a microphone jack/plug, a universalserial bus (USB) port/plug, an Apple® Lightning® port/plug, an Ethernetport/plug, a fiber optic port/plug, a proprietary wired port/plug, aBLUETOOTH® wireless signal transfer, a BLUETOOTH® low energy (BLE)wireless signal transfer, an IBEACON® wireless signal transfer, aradio-frequency identification (RFID) wireless signal transfer,near-field communications (NFC) wireless signal transfer, dedicatedshort range communication (DSRC) wireless signal transfer, 802.11 Wi-Fiwireless signal transfer, wireless local area network (WLAN) signaltransfer, Visible Light Communication (VLC), Worldwide Interoperabilityfor Microwave Access (WiMAX), Infrared (IR) communication wirelesssignal transfer, Public Switched Telephone Network (PSTN) signaltransfer, Integrated Services Digital Network (ISDN) signal transfer,3G/4G/5G/LTE cellular data network wireless signal transfer, ad-hocnetwork signal transfer, radio wave signal transfer, microwave signaltransfer, infrared signal transfer, visible light signal transfer,ultraviolet light signal transfer, wireless signal transfer along theelectromagnetic spectrum, or some combination thereof. Thecommunications interface 1940 may also include one or more GlobalNavigation Satellite System (GNSS) receivers or transceivers that areused to determine a location of the computing system 1900 based onreceipt of one or more signals from one or more satellites associatedwith one or more GNSS systems. GNSS systems include, but are not limitedto, the US-based Global Positioning System (GPS), the Russia-basedGlobal Navigation Satellite System (GLONASS), the China-based BeiDouNavigation Satellite System (BDS), and the Europe-based Galileo GNSS.There is no restriction on operating on any particular hardwarearrangement, and therefore the basic features here may easily besubstituted for improved hardware or firmware arrangements as they aredeveloped.

Storage device 1930 can be a non-volatile and/or non-transitory and/orcomputer-readable memory device and can be a hard disk or other types ofcomputer readable media which can store data that are accessible by acomputer, such as magnetic cassettes, flash memory cards, solid statememory devices, digital versatile disks, cartridges, a floppy disk, aflexible disk, a hard disk, magnetic tape, a magnetic strip/stripe, anyother magnetic storage medium, flash memory, memristor memory, any othersolid-state memory, a compact disc read only memory (CD-ROM) opticaldisc, a rewritable compact disc (CD) optical disc, digital video disk(DVD) optical disc, a blu-ray disc (BDD) optical disc, a holographicoptical disk, another optical medium, a secure digital (SD) card, amicro secure digital (microSD) card, a Memory Stick® card, a smartcardchip, a EMV chip, a subscriber identity module (SIM) card, amini/micro/nano/pico SIM card, another integrated circuit (IC)chip/card, random access memory (RAM), static RAM (SRAM), dynamic RAM(DRAM), read-only memory (ROM), programmable read-only memory (PROM),erasable programmable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cachememory (L1/L2/L3/L4/L5/L#), resistive random-access memory (RRAM/ReRAM),phase change memory (PCM), spin transfer torque RAM (STT-RAM), anothermemory chip or cartridge, and/or a combination thereof.

The storage device 1930 can include software services, servers,services, etc., that when the code that defines such software isexecuted by the processor 1910, it causes the system to perform afunction. In some embodiments, a hardware service that performs aparticular function can include the software component stored in acomputer-readable medium in connection with the necessary hardwarecomponents, such as processor 1910, connection 1905, output device 1935,etc., to carry out the function.

As used herein, the term “computer-readable medium” includes, but is notlimited to, portable or non-portable storage devices, optical storagedevices, and various other mediums capable of storing, containing, orcarrying instruction(s) and/or data. A computer-readable medium mayinclude a non-transitory medium in which data can be stored and thatdoes not include carrier waves and/or transitory electronic signalspropagating wirelessly or over wired connections. Examples of anon-transitory medium may include, but are not limited to, a magneticdisk or tape, optical storage media such as compact disk (CD) or digitalversatile disk (DVD), flash memory, memory or memory devices. Acomputer-readable medium may have stored thereon code and/ormachine-executable instructions that may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a class, or any combination of instructions, datastructures, or program statements. A code segment may be coupled toanother code segment or a hardware circuit by passing and/or receivinginformation, data, arguments, parameters, or memory contents.Information, arguments, parameters, data, etc. may be passed, forwarded,or transmitted using any suitable means including memory sharing,message passing, token passing, network transmission, or the like.

In some embodiments the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Specific details are provided in the description above to provide athorough understanding of the embodiments and examples provided herein.However, it will be understood by one of ordinary skill in the art thatthe embodiments may be practiced without these specific details. Forclarity of explanation, in some instances the present technology may bepresented as including individual functional blocks including functionalblocks comprising devices, device components, steps or routines in amethod embodied in software, or combinations of hardware and software.Additional components may be used other than those shown in the figuresand/or described herein. For example, circuits, systems, networks,processes, and other components may be shown as components in blockdiagram form in order not to obscure the embodiments in unnecessarydetail. In other instances, well-known circuits, processes, algorithms,structures, and techniques may be shown without unnecessary detail inorder to avoid obscuring the embodiments.

Individual embodiments may be described above as a process or methodwhich is depicted as a flowchart, a flow diagram, a data flow diagram, astructure diagram, or a block diagram. Although a flowchart may describethe operations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be re-arranged. A process is terminated when itsoperations are completed, but could have additional steps not includedin a figure. A process may correspond to a method, a function, aprocedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination can correspond to a return of thefunction to the calling function or the main function.

Processes and methods according to the above-described examples can beimplemented using computer-executable instructions that are stored orotherwise available from computer-readable media. Such instructions caninclude, for example, instructions and data which cause or otherwiseconfigure a general purpose computer, special purpose computer, or aprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware,source code, etc. Examples of computer-readable media that may be usedto store instructions, information used, and/or information createdduring methods according to described examples include magnetic oroptical disks, flash memory, USB devices provided with non-volatilememory, networked storage devices, and so on.

Devices implementing processes and methods according to thesedisclosures can include hardware, software, firmware, middleware,microcode, hardware description languages, or any combination thereof,and can take any of a variety of form factors. When implemented insoftware, firmware, middleware, or microcode, the program code or codesegments to perform the necessary tasks (e.g., a computer-programproduct) may be stored in a computer-readable or machine-readablemedium. A processor(s) may perform the necessary tasks. Typical examplesof form factors include laptops, smart phones, mobile phones, tabletdevices or other small form factor personal computers, personal digitalassistants, rackmount devices, standalone devices, and so on.Functionality described herein also can be embodied in peripherals oradd-in cards. Such functionality can also be implemented on a circuitboard among different chips or different processes executing in a singledevice, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are example means for providing the functionsdescribed in the disclosure.

In the foregoing description, aspects of the application are describedwith reference to specific embodiments thereof, but those skilled in theart will recognize that the application is not limited thereto. Thus,while illustrative embodiments of the application have been described indetail herein, it is to be understood that the inventive concepts may beotherwise variously embodied and employed, and that the appended claimsare intended to be construed to include such variations, except aslimited by the prior art. Various features and aspects of theabove-described application may be used individually or jointly.Further, embodiments can be utilized in any number of environments andapplications beyond those described herein without departing from thebroader spirit and scope of the specification. The specification anddrawings are, accordingly, to be regarded as illustrative rather thanrestrictive. For the purposes of illustration, methods were described ina particular order. It should be appreciated that in alternateembodiments, the methods may be performed in a different order than thatdescribed.

One of ordinary skill will appreciate that the less than (“<”) andgreater than (“>”) symbols or terminology used herein can be replacedwith less than or equal to (“<”) and greater than or equal to (“>”)symbols, respectively, without departing from the scope of thisdescription.

Where components are described as being “configured to” perform certainoperations, such configuration can be accomplished, for example, bydesigning electronic circuits or other hardware to perform theoperation, by programming programmable electronic circuits (e.g.,microprocessors, or other suitable electronic circuits) to perform theoperation, or any combination thereof.

The phrase “coupled to” refers to any component that is physicallyconnected to another component either directly or indirectly, and/or anycomponent that is in communication with another component (e.g.,connected to the other component over a wired or wireless connection,and/or other suitable communication interface) either directly orindirectly.

Claim language or other language reciting “at least one of” a set and/or“one or more” of a set indicates that one member of the set or multiplemembers of the set (in any combination) satisfy the claim. For example,claim language reciting “at least one of A and B” or “at least one of Aor B” means A, B, or A and B. In another example, claim languagereciting “at least one of A, B, and C” or “at least one of A, B, or C”means A, B, C, or A and B, or A and C, or B and C, or A and B and C. Thelanguage “at least one of” a set and/or “one or more” of a set does notlimit the set to the items listed in the set. For example, claimlanguage reciting “at least one of A and B” or “at least one of A or B”can mean A, B, or A and B, and can additionally include items not listedin the set of A and B.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein may be implemented as electronic hardware, computer software,firmware, or combinations thereof. To clearly illustrate thisinterchangeability of hardware and software, various illustrativecomponents, blocks, modules, circuits, and steps have been describedabove generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled artisans may implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the present application.

The techniques described herein may also be implemented in electronichardware, computer software, firmware, or any combination thereof. Suchtechniques may be implemented in any of a variety of devices such asgeneral purposes computers, wireless communication device handsets, orintegrated circuit devices having multiple uses including application inwireless communication device handsets and other devices. Any featuresdescribed as modules or components may be implemented together in anintegrated logic device or separately as discrete but interoperablelogic devices. If implemented in software, the techniques may berealized at least in part by a computer-readable data storage mediumcomprising program code including instructions that, when executed,performs one or more of the methods described above. Thecomputer-readable data storage medium may form part of a computerprogram product, which may include packaging materials. Thecomputer-readable medium may comprise memory or data storage media, suchas random access memory (RAM) such as synchronous dynamic random accessmemory (SDRAM), read-only memory (ROM), non-volatile random accessmemory (NVRAM), electrically erasable programmable read-only memory(EEPROM), FLASH memory, magnetic or optical data storage media, and thelike. The techniques additionally, or alternatively, may be realized atleast in part by a computer-readable communication medium that carriesor communicates program code in the form of instructions or datastructures and that can be accessed, read, and/or executed by acomputer, such as propagated signals or waves.

The program code may be executed by a processor, which may include oneor more processors, such as one or more digital signal processors(DSPs), general purpose microprocessors, an application specificintegrated circuits (ASICs), field programmable logic arrays (FPGAs), orother equivalent integrated or discrete logic circuitry. Such aprocessor may be configured to perform any of the techniques describedin this disclosure. A general purpose processor may be a microprocessor;but in the alternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration. Accordingly, the term “processor,” as used herein mayrefer to any of the foregoing structure, any combination of theforegoing structure, or any other structure or apparatus suitable forimplementation of the techniques described herein.

Illustrative aspects of the disclosure include:

Aspect 1: A method for processing one or more frames, comprising:obtaining a first plurality of frames associated with a first settingsdomain from an image capture system, wherein the first plurality offrames is captured prior to obtaining a capture input; obtaining atleast one reference frame associated with a second settings domain fromthe image capture system, wherein the at least one reference frame iscaptured proximate to obtaining the capture input; obtaining a secondplurality of frames associated with the second settings domain from theimage capture system, wherein the second plurality of frames is capturedafter the at least one reference frame; and based on the at least onereference frame, transforming at least a portion of the first pluralityof frames to generate a transformed plurality of frames associated withthe second settings domain.

Aspect 2: The method of Aspect 1, wherein: the first settings domaincomprises a first resolution; the second settings domain comprises asecond resolution; and transforming at least the portion of the firstplurality of frames comprises upscaling at least the portion of thefirst plurality of frames from the first resolution to the secondresolution to generate the transformed plurality of frames, wherein thetransformed plurality of frames have the second resolution.

Aspect 3: The method of any of Aspects 1 to 2, further comprising:obtaining an additional reference frame having the second resolutionfrom the image capture system, wherein the additional reference frame iscaptured prior to obtaining the capture input, wherein generating theupscaled plurality of frames having the second resolution is based on atleast the portion of the first plurality of frames, the at least onereference frame, and the additional reference frame, and wherein the atleast one reference frame provides a reference for upscaling at least afirst portion of at least the portion of the first plurality of framesand the additional reference frame provides a reference for upscaling atleast a second portion of at least the portion of the first plurality offrames.

Aspect 4: The method of any of Aspects 1 to 3, further comprising:combining the transformed plurality of frames and the second pluralityof frames to generate a video associated with the second settingsdomain.

Aspect 5: The method of any of Aspects 1 to 4, further comprising:obtaining motion information associated with the first plurality offrames, wherein generating the transformed plurality of framesassociated with the second settings domain is based on at least theportion of the first plurality of frames, the at least one referenceframe, and the motion information.

Aspect 6: The method of any of Aspects 1 to 5, further comprising:determining a panning direction based on the motion informationassociated with the first plurality of frames; and applying the panningdirection to the transformed plurality of frames.

Aspect 7: The method of any of Aspects 1 to 6, wherein: the firstsettings domain comprises a first framerate; the second settings domaincomprises a second framerate; and transforming at least the portion ofthe first plurality of frames comprises framerate converting at leastthe portion of the first plurality of frames from the first framerate tothe second framerate.

Aspect 8: The method of any of Aspects 1 to 7, wherein a first subset ofthe first plurality of frames is captured at the first framerate and asecond subset of the first plurality of frames is captured at a thirdframerate, different from the first framerate, wherein the thirdframerate is equal to or not equal to the second framerate, and whereina change between the first framerate and the third framerate is based atleast in part on motion information associated at least one of the firstsubset of the first plurality of frames and the second subset of thefirst plurality of frames.

Aspect 9: The method of any of Aspects 1 to 8, wherein: the firstsettings domain comprises a first resolution and a first framerate; thesecond settings domain comprises a second resolution and a secondframerate; and transforming at least the portion of the first pluralityof frames comprises upscaling at least the portion of the firstplurality of frames from the first resolution to the second resolutionand framerate converting at least the portion of the first plurality offrames from the first framerate to the second framerate.

Aspect 10: The method of any of Aspects 1 to 9, further comprising:obtaining an additional reference frame associated with the secondsettings domain from the image capture system, wherein the additionalreference frame is captured prior to obtaining the capture input,wherein generating the transformed plurality of frames associated withthe second settings domain is based on at least the portion of the firstplurality of frames, at least one reference frame, and the additionalreference frame, and wherein the at least one reference frame provides areference for transforming at least a first subset of at least theportion of the first plurality of frames and the additional referenceframe provides a reference for transforming at least a second subset ofat least the portion of the first plurality of frames.

Aspect 11: The method of any of Aspects 1 to 10, wherein the at leastone reference frame comprises a first reference frame, the methodfurther comprising: obtaining a second reference frame associated withthe second settings domain from the image capture system, wherein thesecond reference frame is captured proximate to obtaining the captureinput; based on the first reference frame, transforming at least theportion of the first plurality of frames to generate the transformedplurality of frames associated with the second settings domain; andbased on the second reference frame, transforming at least anotherportion of the first plurality of frames to generate a secondtransformed plurality of frames associated with the second settingsdomain.

Aspect 12: The method of any of Aspects 1 to 11, further comprising:obtaining a motion estimate associated with the first plurality offrames; obtaining a third reference frame associated with the secondsettings domain from the image capture system, wherein the thirdreference frame is captured prior to obtaining the capture input; andbased on the third reference frame, transforming a third portion of thefirst plurality of frames to generate a third transformed plurality offrames associated with the second settings domain; wherein an amount oftime between the first reference frame and the third reference frame isbased on the motion estimate associated with the first plurality offrames.

Aspect 13: The method of any of Aspects 1 to 12, wherein the firstsettings domain comprises at least one of a first resolution, a firstframerate, a first color depth, a first noise reduction technique, afirst edge enhancement technique, a first image stabilization technique,and a first color correction technique; and the second settings domaincomprises at least one of a second resolution, a second framerate, asecond color depth, a second noise reduction technique, a second edgeenhancement technique, a second image stabilization technique, and asecond color correction technique.

Aspect 14: The method of any of Aspects 1 to 13, further comprising:generating the transformed plurality of frames using a trainable neuralnetwork, wherein the neural network is trained using a training datasetcomprising pairs of images, each pair of images including a first imageassociated with the first settings domain and a second image associatedwith the second settings domain.

Aspect 15: The method of any of Aspects 1 to 14, wherein capturing theat least one reference frame proximate to obtaining the capture inputcomprises capturing a first available associated with the secondsettings domain after the capture input is received, capturing a secondavailable frame associated with the second settings domain after thecapture input is received, capturing a third available frame associatedwith the second settings domain after the capture input is received, orcapturing a fourth available frame associated with the second settingsdomain after the capture input is received.

Aspect 16: The method of any of Aspects 1 to 15, wherein capturing theat least one reference frame proximate to obtaining the capture inputcomprises capturing a frame associated with the second settings domainwithin 10 millisecond (ms), within 100 ms, within 500 ms, or within 1000ms after the capture input is received.

Aspect 17: An apparatus for processing one or more frames, comprising:at least one memory; and one or more processors coupled to the at leastone memory and configured to: obtain a first plurality of framesassociated with a first settings domain from an image capture system,wherein the first plurality of frames is captured prior to obtaining acapture input, obtain at least one reference frame associated with asecond settings domain from the image capture system, wherein the atleast one reference frame is captured proximate to obtaining the captureinput, obtain a second plurality of frames associated with the secondsettings domain from the image capture system, wherein the secondplurality of frames is captured after the at least one reference frame,and based on the at least one reference frame, transform at least aportion of the first plurality of frames to generate a transformedplurality of frames associated with the second settings domain.

Aspect 18: The apparatus of Aspect 17, wherein: the first settingsdomain comprises a first resolution; the second settings domaincomprises a second resolution; and, to transform at least the portion ofthe first plurality of frames, the one or more processors are configuredto upscale at least the portion of the first plurality of frames fromthe first resolution to the second resolution to generate thetransformed plurality of frames, wherein the transformed plurality offrames have the second resolution.

Aspect 19: The apparatus of any of Aspects 17 to 18, wherein the one ormore processors are configured to: obtain an additional reference framehaving the second resolution from the image capture system, wherein theadditional reference frame is captured prior to obtaining the captureinput, wherein generating the upscaled plurality of frames having thesecond resolution is based on at least the portion of the firstplurality of frames, the at least one reference frame, and theadditional reference frame, and wherein the at least one reference frameprovides a reference for upscaling at least a first portion of at leastthe portion of the first plurality of frames and the additionalreference frame provides a reference for upscaling at least a secondportion of at least the portion of the first plurality of frames.

Aspect 20: The apparatus of any of Aspects 17 to 19, wherein the one ormore processors are configured to: combine the transformed plurality offrames and the second plurality of frames to generate a video associatedwith the second settings domain.

Aspect 21: The apparatus of any of Aspects 17 to 20, wherein the one ormore processors are configured to: obtain motion information associatedwith the first plurality of frames, wherein generating the transformedplurality of frames associated with the second settings domain is basedon at least the portion of the first plurality of frames, the at leastone reference frame, and the motion information.

Aspect 22: The apparatus of any of Aspects 17 to 21, wherein the one ormore processors are configured to: determine a panning direction basedon the motion information associated with the first plurality of frames;and apply the panning direction to the transformed plurality of frames.

Aspect 23: The apparatus of any of Aspects 17 to 22, wherein: the firstsettings domain comprises a first framerate; the second settings domaincomprises a second framerate; and to transform at least the portion ofthe first plurality of frames, the one or more processors are configuredto framerate convert at least the portion of the first plurality offrames from the first framerate to the second framerate.

Aspect 24: The apparatus of any of Aspects 17 to 23, wherein a firstsubset of the first plurality of frames is captured at the firstframerate and a second subset of the first plurality of frames iscaptured at a third framerate, different from the first framerate,wherein the third framerate is equal to or not equal to the secondframerate, and wherein a change between the first framerate and thethird framerate is based at least in part on motion informationassociated at least one of the first subset of the first plurality offrames and the second subset of the first plurality of frames.

Aspect 25: The apparatus of any of Aspects 17 to 24, wherein: the firstsettings domain comprises a first resolution and a first framerate; thesecond settings domain comprises a second resolution and a secondframerate; and to transform at least the portion of the first pluralityof frames, the one or more processors are configured to upscale at leastthe portion of the first plurality of frames from the first resolutionto the second resolution and framerate convert at least the portion ofthe first plurality of frames from the first framerate to the secondframerate.

Aspect 26: The apparatus of any of Aspects 17 to 25, wherein the one ormore processors are configured to: obtain an additional reference frameassociated with the second settings domain from the image capturesystem, wherein the additional reference frame is captured prior toobtaining the capture input, generating the transformed plurality offrames associated with the second settings domain is based on at leastthe portion of the first plurality of frames, at least one referenceframe, and the additional reference frame, and wherein the at least onereference frame provides a reference for transforming at least a firstsubset of at least the portion of the first plurality of frames and theadditional reference frame provides a reference for transforming atleast a second subset of at least the portion of the first plurality offrames.

Aspect 27: The apparatus of any of Aspects 17 to 26, wherein the one ormore processors are configured to: obtain a second reference frameassociated with the second settings domain from the image capturesystem, wherein the second reference frame is captured proximate toobtaining the capture input; based on the first reference frame,transform at least the portion of the first plurality of frames togenerate the transformed plurality of frames associated with the secondsettings domain; and based on the second reference frame, transform atleast another portion of the first plurality of frames to generate asecond transformed plurality of frames associated with the secondsettings domain.

Aspect 28: The apparatus of any of Aspects 17 to 27, wherein the one ormore processors are configured to: obtain a motion estimate associatedwith the first plurality of frames; and obtain a third reference frameassociated with the second settings domain from the image capturesystem, wherein the third reference frame is captured prior to obtainingthe capture input; and based on the third reference frame, transform athird portion of the first plurality of frames to generate a thirdtransformed plurality of frames associated with the second settingsdomain; wherein an amount of time between the first reference frame andthe third reference frame is based on the motion estimate associatedwith the first plurality of frames.

Aspect 29: The apparatus of any of Aspects 17 to 28, wherein the firstsettings domain comprises at least one of a first resolution, a firstframerate, a first color depth, a first noise reduction technique, afirst edge enhancement technique, a first image stabilization technique,and a first color correction technique, and the second settings domaincomprises at least one of a second resolution, a second framerate, asecond color depth, a second noise reduction technique, a second edgeenhancement technique, a second image stabilization technique, and asecond color correction technique.

Aspect 30: The apparatus of any of Aspects 17 to 29, wherein the one ormore processors are configured to: generate the transformed plurality offrames use a trainable neural network, wherein the neural network istrained using a training dataset comprising pairs of images, each pairof images including a first image associated with the first settingsdomain and a second image associated with the second settings domain.

Aspect 31: The apparatus of any of Aspects 17 to 30, wherein, to capturethe at least one reference frame proximate to obtaining the captureinput, the one or more processors are configured to: capture a firstavailable associated with the second settings domain after the captureinput is received; capture a second available frame associated with thesecond settings domain after the capture input is received; capture athird available frame associated with the second settings domain afterthe capture input is received; or capture a fourth available frameassociated with the second settings domain after the capture input isreceived.

Aspect 32: The apparatus of any of Aspects 17 to 31, wherein, to capturethe at least one reference frame proximate to obtaining the captureinput, the one or more processors are configured to capture a frameassociated with the second settings domain within 10 millisecond (ms),within 100 ms, within 500 ms, or within 1000 ms after the capture inputis received.

Aspect 33: A non-transitory computer-readable storage medium havingstored thereon instructions which, when executed by one or moreprocessors, cause the one or more processors to perform any of theoperations of aspects 1 to 32.

Aspect 34: An apparatus comprising means for performing any of theoperations of aspects 1 to 32.

Aspect 35: A method for processing one or more frames comprising:obtaining a first plurality of frames associated with a first settingsdomain from an image capture system, wherein the first plurality offrames is captured prior to obtaining a capture input; obtaining areference frame associated with a second settings domain from the imagecapture system, wherein the reference frame is captured proximate toobtaining the capture input; obtaining a selection of one or moreselected frames associated with the first plurality of frames; and basedon the reference frame, transforming the one or more selected frames togenerate one or more transformed frames associated with the secondsettings domain.

Aspect 36: The method of Aspect 35, wherein selection of one or moreselected frames is based on a selection from a user interface.

Aspect 37: The method of any of Aspects 35 to 36, wherein the userinterface comprises a thumbnail gallery, a slider, or a frame-by-framereview.

Aspect 38: An apparatus for processing one or more frames, comprising:at least one memory; and one or more processors coupled to the at leastone memory and configured to: obtain a first plurality of framesassociated with a first settings domain from an image capture system,wherein the first plurality of frames is captured prior to obtaining acapture input, obtain a reference frame associated with a secondsettings domain from the image capture system, wherein the referenceframe is captured proximate to obtaining the capture input, obtain aselection of one or more selected frames associated with the firstplurality of frames, and based on the reference frame, transform the oneor more selected frames to generate one or more transformed framesassociated with the second settings domain.

Aspect 39: The apparatus of Aspect 38, wherein selection of one or moreselected frames is based on a selection from a user interface.

Aspect 40: The apparatus of any of Aspects 38 to 39, wherein the userinterface comprises a thumbnail gallery, a slider, or a frame-by-framereview.

Aspect 41: A non-transitory computer-readable storage medium havingstored thereon instructions which, when executed by one or moreprocessors, cause the one or more processors to perform any of theoperations of aspects 35 to 40.

Aspect 42: An apparatus comprising means for performing any of theoperations of aspects 35 to 40.

Aspect 43: A method comprising operations according to any of Aspects 1to 32 and any of Aspects 35 to 40.

Aspect 44: An apparatus for processing one or more frames. The apparatusincludes at least one memory (e.g., implemented in circuitry) configuredto store one or more frames and one or more processors (e.g., oneprocessor or multiple processors) coupled to the at least one memory.The one or more processors are configured to perform operationsaccording to any of Aspects 1 to 32 and any of Aspects 35-40.

Aspect 45: A computer-readable storage medium storing instructions that,when executed by one or more processors, cause the one or moreprocessors to perform operations according to any of Aspects 26 to 35and any of Aspects 36 to 43.

Aspect 46: An apparatus comprising means for performing operationsaccording to any of Aspects 26 to 37 and any of Aspects 38 to 45.

What is claimed is:
 1. A method for processing one or more frames,comprising: obtaining a first plurality of frames from an image capturesystem, wherein the first plurality of frames is captured prior toobtaining a capture input; obtaining at least one additional frame inresponse to the capture input; displaying an interface for selecting atleast one selected frame, each selected frame associated with one ormore frames of the first plurality of frames; receiving, by theinterface, a selection of the at least one selected frame; and based onthe selection, generating an image based on the at least one selectedframe and the at least one additional frame.
 2. The method of claim 1,wherein the at least one additional frame is obtained from an additionalimage capture system different from the image capture system.
 3. Themethod of claim 1, wherein the interface comprises one or more of aslider for advancing through frames or a gallery corresponding to thefirst plurality of frames.
 4. The method of claim 1, wherein the firstplurality of frames is stored in a frame buffer.
 5. The method of claim1, further comprising determining at least one additional selected frameassociated with one or more frames of the first plurality of frames,wherein determining the at least one additional selected frame comprisesone or more of: determining a change of the at least one additionalselected frame relative to one or more frames captured in temporalproximity to the at least one additional selected frame; determiningthat an amount of motion in the at least one additional selected frameexceeds a motion threshold; determining that the at least one additionalframe contains relevant content; determining that the at least oneadditional frame contains a human face; or processing the at least oneadditional selected frame by a neural network.
 6. The method of claim 1,wherein displaying the interface comprises displaying at least onesuggested frame.
 7. The method of claim 6, further comprising:determining a change of the at least one suggested frame relative to oneor more frames captured in temporal proximity to the at least onesuggested frame; and displaying the at least one suggested frame basedon determining the change of the at least one suggested frame.
 8. Themethod of claim 6, further comprising: determining that an amount ofmotion in the at least one suggested frame exceeds a motion threshold;and displaying the at least one suggested frame based on determiningthat the amount of motion in the at least one suggested frame exceedsthe motion threshold.
 9. The method of claim 6, wherein the at least onesuggested frame is generated using a deep learning neural network. 10.The method of claim 1, wherein: the at least one selected frame isassociated with a first settings domain; the at least one additionalframe is associated with a second settings domain; and generating theimage comprises transforming the at least one selected frame from thefirst settings domain to the second settings domain.
 11. The method ofclaim 10, wherein: the first settings domain comprises a firstresolution; the second settings domain comprises a second resolution;and transforming the at least one selected frame from the first settingsdomain to the second settings domain comprises upscaling the at leastone selected frame from the first resolution to the second resolution.12. The method of claim 10, wherein: the first settings domain comprisesa first framerate; the second settings domain comprises a secondframerate; and transforming the at least one selected frame from thefirst settings domain to the second settings domain comprises framerateconverting the at least one selected frame from the first framerate tothe second framerate.
 13. The method of claim 10, wherein: the firstsettings domain comprises a first resolution and a first framerate; thesecond settings domain comprises a second resolution and a secondframerate; and transforming the at least one selected frame from thefirst settings domain to the second settings domain comprises upscalingthe at least one selected frame from the first resolution to the secondresolution and framerate converting the at least one selected frame fromthe first framerate to the second framerate.
 14. The method of claim 1,wherein displaying the interface for selecting the at least one selectedframe comprises displaying one or more frames of the first plurality offrames.
 15. The method of claim 1, wherein the at least one selectedframe includes a frame selected from the first plurality of frames. 16.The method of claim 1, wherein the at least one additional frame is akeyframe.
 17. An apparatus for processing one or more frames,comprising: a memory; and one or more processors coupled to the memoryand configured to: obtain a first plurality of frames from an imagecapture system, wherein the first plurality of frames is captured priorto obtaining a capture input; obtain at least one additional frame inresponse to the capture input; display an interface for selecting atleast one selected frame, each selected frame associated with one ormore frames of the first plurality of frames; receive, by the interface,a selection of the at least one selected frame; and based on theselection, generate an image based on the at least one selected frameand the at least one additional frame.
 18. The apparatus of claim 17,wherein the at least one additional frame is obtained from an additionalimage capture system different from the image capture system.
 19. Theapparatus of claim 17, wherein the interface comprises one or more of aslider for advancing through frames or a gallery corresponding to thefirst plurality of frames.
 20. The apparatus of claim 17, whereindisplaying the interface for selecting the at least one selected framecomprises displaying one or more frames of the first plurality offrames.
 21. The apparatus of claim 17, wherein, to display theinterface, the one or more processors are configured to display at leastone suggested frame.
 22. The apparatus of claim 21, further comprising:determining a change of the at least one suggested frame relative to oneor more frames captured in temporal proximity to the at least onesuggested frame; and displaying the at least one suggested frame basedon determining the change of the at least one suggested frame.
 23. Theapparatus of claim 21, further comprising: determining that an amount ofmotion in the at least one suggested frame exceeds a motion threshold;and displaying the at least one suggested frame based on determiningthat the amount of motion in the at least one suggested frame exceedsthe motion threshold.
 24. The apparatus of claim 21, wherein the atleast one suggested frame is generated using a deep learning neuralnetwork.
 25. The apparatus of claim 17, wherein the at least oneselected frame includes a frame selected from the first plurality offrames.
 26. The apparatus of claim 17, wherein the at least oneadditional frame is a keyframe.
 27. An apparatus for processing one ormore frames, comprising: a display; an image capture system; a memory;and one or more processors coupled to the memory and configured to:obtain a first plurality of frames from the image capture system,wherein the first plurality of frames is captured prior to obtaining acapture input; obtain at least one additional frame in response to thecapture input; display, by the display, an interface for selecting atleast one selected frame, each selected frame associated with one ormore frames of the first plurality of frames; receive, by the interface,a selection of the at least one selected frame; and based on theselection, generate an image based on the at least one selected frameand the at least one additional frame.
 28. The apparatus of claim 27,wherein the at least one additional frame is obtained from an additionalimage capture system different from the image capture system.
 29. Theapparatus of claim 27, wherein, to display the interface, the one ormore processors are configured to display, by the display, at least onesuggested frame.
 30. The apparatus of claim 27, wherein the at least oneadditional frame is a keyframe.